Launching tests¶
pytest¶
Since we use pytest to run the tests, you can also use it on its own. We do recommend using the tox command though:
pytest tests/ -k test_functional
You can use pytest directly. If you want to run tests on a specific portion of the code with pytest and your local python version:
python3 -m pytest
Everything in tests/message with coverage for the relevant code (require pytest-cov):
python3 -m pytest tests/message/ --cov=pylint.message
coverage html
Only the functional test "missing_kwoa_py3":
python3 -m pytest "tests/test_functional.py::test_functional[missing_kwoa_py3]"
tox¶
You can also optionally install tox and run our tests using the tox package, as in:
python -m tox
python -m tox -epy38 # for Python 3.8 suite only
python -m tox -epylint # for running Pylint over Pylint's codebase
python -m tox -eformatting # for running formatting checks over Pylint's codebase
It's usually a good idea to run tox with --recreate
. This flag tells tox to re-download
all dependencies before running the tests. This can be important when a new version of
astroid or any of the other dependencies has been published:
python -m tox --recreate # The entire tox environment will be recreated
python -m tox --recreate -e py310 # The python 3.10 tox environment will be recreated
To run only a specific test suite, use a pattern for the test filename
(without the .py
extension), as in:
python -m tox -e py310 -- -k test_functional
python -m tox -e py310 -- -k \*func\*
python -m tox --recreate -e py310 -- -k test_functional # With recreation of the environment
Primer tests¶
Pylint also uses what we refer to as primer
tests. These are tests that are run automatically
in our Continuous Integration and check whether any changes in Pylint lead to crashes or fatal errors
on the stdlib
, and also assess a pull request's impact on the linting of a selection of external
repositories by posting the diff against pylint
's current output as a comment.
To run the primer test for the stdlib
, which only checks for crashes and fatal errors, you can add
--primer-stdlib
to the pytest command. For example:
pytest -m primer_stdlib --primer-stdlib
To produce the output generated on Continuous Integration for the linting of external repositories, run these commands:
python tests/primer/__main__.py prepare --clone
python tests/primer/__main__.py run --type=pr
To fully simulate the process on Continuous Integration, you should then checkout main
, and
then run these commands:
python tests/primer/__main__.py run --type=main
python tests/primer/__main__.py compare
The list of repositories is created on the basis of three criteria: 1) projects need to use a diverse range of language features, 2) projects need to be well maintained and 3) projects should not have a codebase that is too repetitive. This guarantees a good balance between speed of our CI and finding potential bugs.
You can find the latest list of repositories and any relevant code for these tests in the tests/primer
directory.