I use poetry as my python package manager and I'd like to know the packages that I depend on that are currently outdated.
An easy way to get this list is to run
poetry show --outdated. This will return you a list of all the packages that are outdated, their current version, the latest version, as well as a description of the package.
There are in my opinion three missing features here:
- having the command respect the semantic versioning constraint and only letting you see the latest version according to those constraints
- having a flag to switch between showing the latest version available without semantic versioning constraint vs the latest version constrained by semantic versioning
- having a flag to list only the packages that are direct dependencies (listed in the
I use mypy but it doesn't seem to scan all my files. Why?
You might be using implicit namespaces in your code (see PEP 420). Support for implicit namespaces in
mypy is rather flaky as of 2020-03-03.
One solution for the moment is to add
__init__.py and make all your namespaces explicit.
Another solution is to replace your calls to
mypy some-path with
mypy $(find some-path -name "*.py").
Some imports in my python code are slow. How can I figure out which ones are the source of slowness?
Python offers a really useful functionality you can use that will list how long each import took. By passing the
-X importtime argument to your python command when you execute your script it will print out both the cumulative time (including nested imports) and self time (excluding nested imports) of each import.
python -X importtime your-script.py
python -X importtime my-script.py on an empty script returns the following (on Windows 7, Python 3.7.5)
import time: self [us] | cumulative | imported package import time: 52 | 52 | zipimport import time: 367 | 367 | _frozen_importlib_external import time: 55 | 55 | _codecs import time: 530 | 585 | codecs import time: 520 | 520 | encodings.aliases import time: 1107 | 2210 | encodings import time: 328 | 328 | encodings.utf_8 import time: 39 | 39 | _signal import time: 357 | 357 | encodings.latin_1 import time: 34 | 34 | _abc import time: 312 | 345 | abc import time: 474 | 819 | io import time: 113 | 113 | _stat import time: 264 | 377 | stat import time: 269 | 269 | genericpath import time: 794 | 1062 | ntpath import time: 871 | 871 | _collections_abc import time: 915 | 3223 | os import time: 490 | 490 | _sitebuiltins import time: 57 | 57 | _locale import time: 934 | 991 | _bootlocale import time: 421 | 421 | encodings.cp1252 import time: 472 | 472 | types import time: 394 | 394 | warnings import time: 440 | 834 | importlib import time: 263 | 263 | importlib.machinery import time: 554 | 816 | importlib.abc import time: 64 | 64 | _operator import time: 792 | 856 | operator import time: 343 | 343 | keyword import time: 43 | 43 | _heapq import time: 405 | 447 | heapq import time: 85 | 85 | itertools import time: 328 | 328 | reprlib import time: 69 | 69 | _collections import time: 1460 | 3585 | collections import time: 44 | 44 | _functools import time: 596 | 640 | functools import time: 784 | 5008 | contextlib import time: 651 | 7308 | importlib.util import time: 1095 | 1095 | pywin32_bootstrap import time: 231 | 231 | sitecustomize import time: 10744 | 24972 | site
For a script with a simple
import argparse, I get the following output:
import time: self [us] | cumulative | imported package import time: 70 | 70 | zipimport import time: 341 | 341 | _frozen_importlib_external import time: 54 | 54 | _codecs import time: 457 | 511 | codecs import time: 456 | 456 | encodings.aliases import time: 1030 | 1997 | encodings import time: 215 | 215 | encodings.utf_8 import time: 38 | 38 | _signal import time: 268 | 268 | encodings.latin_1 import time: 33 | 33 | _abc import time: 398 | 431 | abc import time: 311 | 741 | io import time: 87 | 87 | _stat import time: 271 | 357 | stat import time: 196 | 196 | genericpath import time: 416 | 612 | ntpath import time: 714 | 714 | _collections_abc import time: 610 | 2292 | os import time: 229 | 229 | _sitebuiltins import time: 48 | 48 | _locale import time: 246 | 293 | _bootlocale import time: 217 | 217 | encodings.cp1252 import time: 488 | 488 | types import time: 279 | 279 | warnings import time: 461 | 740 | importlib import time: 269 | 269 | importlib.machinery import time: 557 | 825 | importlib.abc import time: 63 | 63 | _operator import time: 808 | 871 | operator import time: 336 | 336 | keyword import time: 41 | 41 | _heapq import time: 336 | 376 | heapq import time: 69 | 69 | itertools import time: 341 | 341 | reprlib import time: 70 | 70 | _collections import time: 1136 | 3197 | collections import time: 69 | 69 | _functools import time: 642 | 710 | functools import time: 801 | 4708 | contextlib import time: 688 | 6959 | importlib.util import time: 934 | 934 | pywin32_bootstrap import time: 224 | 224 | sitecustomize import time: 9323 | 20954 | site import time: 698 | 698 | enum import time: 55 | 55 | _sre import time: 417 | 417 | sre_constants import time: 372 | 789 | sre_parse import time: 443 | 1286 | sre_compile import time: 323 | 323 | copyreg import time: 716 | 3021 | re import time: 725 | 725 | locale import time: 948 | 1673 | gettext import time: 986 | 5678 | argparse
The package are listed in order that they are resolved. In
sys were already loaded, so it first loads
gettext. Once both are loaded,
argparse has finished loading.
The way the cumulative column is computed is to take all the prior self that are a level higher than the package you're looking at. For example (if we take the io package):
import time: 33 | 33 | _abc import time: 398 | 431 | abc import time: 311 | 741 | io
311 + 398 + 33 = 742
We can see here that the numbers are not necessarily equal to one another, this might be due to precision used to do the computation while the rendering of numbers is rounded.
Note that the load time of a package may be different depending on which script you load because dependendencies of the package may have already been loaded in some cases, while in others it may have to load them.
Looking at text might be your thing, but if you're more visual, there's a tool called tuna which will consume this output and create an icicle plot you can look at to find which imports are the slowest/longest.
My click CLI is slow, even just to show the help. How do I make it go faster?
In most cases, the reason your click CLI is slow is that you have large imports at the top of the files where you have declared your commands.
The typical pattern is as follows:
from train import train from predict import predict @click.group() def cli(): pass cli.add_command(predict) cli.add_command(train)
import click import pandas as pd import torch @click.command() def train(): pass
import click import pandas as pd import torch @click.command() def predict(): pass
Notice that in both these files we import
torch, which can account for a large chunk of script execution time simply due to importing them. You can verify that by simply running
python -X importtime train.py 2>tuna.log and using tuna (run
tuna tuna.log) to inspect the results and convince yourself.
The suggested pattern is to move the imports inside of the function itself, as such:
import click @click.command() def train(): import pandas as pd import torch pass
import click @click.command() def predict(): import pandas as pd import torch pass
This will shave off a large amount of time spent importing those packages (
torch). They will only be loaded when you need to run the command itself, not every time you invoke the CLI.
Another pattern which is more complicated is to move the logic of the functions in separate files. This is done to avoid the common mistake that will happen over time that developers will add more logic in those command files, adding imports at the top of the file and slowing the CLI again. By moving the complete implementation to a separate file, you can have the imports at the top of the file and it is not possible to make this mistake again.
import click @click.command() def train(): from train_implementation import train train()
import pandas as pd import torch def train(): # Implementation is now here pass
I have two test files with the same name and pytest complains. How do I make it work without changing the test filenames?
Example directory structure
/path/to/project/tests ├── a/ │ └── test_a.py └── b/ └── test_a.py
import file mismatch: imported module 'test_a' has this __file__ attribute: /path/to/project/tests/a/test_a.py which is not the same as the test file we want to collect: /path/to/project/tests/b/test_a.py HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
__init__.py to each directories with tests files that have the same name. Technically, you only need to have a
__init__.py file in one of the two directories, so that one is in a package while the other one is in a different one. Adding it in both simply prevents this issue from occurring again if you were to add a third file
/path/to/project/tests ├── a/ │ ├── __init__.py │ └── test_a.py └── b/ ├── __init__.py └── test_a.py