Pythons packaging

How to create a simple python package

python logo

What are we going to do?

  1. Create a package (with a setup.py script)
  2. Make it available on PyPI

What does this do?

  1. Allow us to easily install a python script from a distribution
    python3 setup.py install
    
  2. Or from Internet
    pip install package_name
    

What are we going to use?

  • Python 3 (it's nearly the same for python 2)
  • distutils (packaging framework)
  • setuptools (Python Packaging Authority enhenced alternative)
  • PyPI (Python Package Index)

Directory_Structure

Simple mode

With a very simple tool that have only 1 python script file, the .py script file can be placed under the root folder.

└── tool-name
    ├── tool-name.py
    ├── LICENCE
    ├── README.md
    └── setup.py

Lib mode

But with a more complex tool with several python script files, .py script files must be place under a "lib" folder.

└── tool-name
    ├── tool-name
    │   ├── __init__.py
    │   ├── script.py
    │   └── main.py
    ├── LICENCE
    ├── README.md
    └── setup.py

And of course a __init__.py will be needed.

Setup-Script

Description

  • The distutils setup script is a python script!
  • Keep it simple to avoid bug happening, exoticism is not needed.
  • It describes project's metadatas

First line

First line of distutils setup script will always be the same:

from distutils.core import setup

This will import setup() function that will take some parameters.

Minimal parameters

Only 3 field are required:

  • name: must be unique to publish on PyPI
  • version: keeps track of different releases
  • url: the home page address of the project (ex: github, bitbucket, website, PyPI page, ...)

More basic parameters (1/3)

  • description: one-line summary of the project
  • long_description: multi-line string in reStructuredText (PyPI will converts it to HTML and displays it on the package page)
  • author: author name
  • author_email: author email

More basic parameters (2/3)

  • maintainer: maintainer name
  • maintainer_email: maintainer email
  • license: license name
  • keywords: keywords
  • classifiers: list of PyPI categories (more details in the next slides)

More basic parameters (3/3)

An a vey important field in order to differentiate the source code and all the others things around like documentation:

  • py_modules or packages: describes where python source code is
    • py_modules: for an unique file python script
    • packages: for a multi-file python script

Additional meta-data can be found in the Python Doc.

There is also a PEP 314 describing a mechanism for adding metadata to Python packages.

Classifying the package

PyPI (formerly Python Package Index) contains thousands of Python libraries and tools. So a proper classification of metada will allow people to easily find the package.

To classify the package, use the classifiers parameter of the setup() function. This contains a list of strings.

All classifier strings must match with PyPi classifiers list.

Setup script example

from distutils.core import setup
setup(
    name = "chardet",
    packages = ["chardet"],
    version = "1.0.2",
    description = "Universal encoding detector",
    author = "Mark Pilgrim",
    author_email = "mark@diveintomark.org",
    url = "http://chardet.feedparser.org/",
    download_url = "http://chardet.feedparser.org/download/python3-chardet-1.0.1.tgz",
    keywords = ["encoding", "i18n", "xml"],
    classifiers = [
        "Programming Language :: Python",
        "Programming Language :: Python :: 3",
        "Development Status :: 4 - Beta",
        "Environment :: Other Environment",
        "Intended Audience :: Developers",
        "License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)",
        "Operating System :: OS Independent",
        "Topic :: Software Development :: Libraries :: Python Modules",
        "Topic :: Text Processing :: Linguistic",
        ],
    long_description = """\
Universal character encoding detector
-------------------------------------

Detects
 - ASCII, UTF-8, UTF-16 (2 variants), UTF-32 (4 variants)
 - Big5, GB2312, EUC-TW, HZ-GB-2312, ISO-2022-CN (Traditional and Simplified Chinese)
 - EUC-JP, SHIFT_JIS, ISO-2022-JP (Japanese)
 - EUC-KR, ISO-2022-KR (Korean)
 - KOI8-R, MacCyrillic, IBM855, IBM866, ISO-8859-5, windows-1251 (Cyrillic)
 - ISO-8859-2, windows-1250 (Hungarian)
 - ISO-8859-5, windows-1251 (Bulgarian)
 - windows-1252 (English)
 - ISO-8859-7, windows-1253 (Greek)
 - ISO-8859-8, windows-1255 (Visual and Logical Hebrew)
 - TIS-620 (Thai)

This version requires Python 3 or later; a Python 2 version is available separately.
"""
)

Manifest

Specifying additional files with a Manifest (1/2)

By default, Distutils will only include the following files in the release package:

  • README.txt
  • setup.py
  • files listed in packages or py_modules parameters

So to include a LICENCE file, a NOTICE file, use a different file extension like a README.md or docs/ documentation folder, a Manifest file will be needed.

Specifying additional files with a Manifest (2/2)

The Manifest file is called MANIFEST.in, placed in the project's root directory. This is not a python script. It allows to include or exclude files and directories.

include README.md
recursive-include docs *.html *.css

It maintains the directory structure.

Manifest format and commands.

Setup.cfg

setup.cfg is a powerful tool that can do a lot but here we only need it because we're using a markdown readme file. Because Distutils only check for README and README.txt.

[metadata]
description-file = README.md

Checking the setup script

Distutils has a built-in validation command:

python3 setup.py check
running check

Creating a source distribution

The minimum is too build a source distribution containing the source code, the setup script, the README file and additional files that come with the MANIFEST.

python3 setup.py sdist

This will create a dist/ folder in your project directory and a package_name.tar.gz archive inside it (or a .zip for Windows users).

Creating_dumb_built_distributions

Creating dumb built distributions (1/2)

Python docs

RPM packages

python setup.py bdist_rpm

or

python setup.py bdist --formats=rpm

Creating dumb built distributions (2/2)

Windows Installers

python setup.py bdist_wininst

or

python setup.py bdist --formats=wininst

Submit-to-PyPI

PyPI

PyPI is the Python Package Index

  • Easy distribution
  • Doesn't need cloning and manual building
pip install something

Create a .pypirc configuration file (1/2)

To make life easier when uploading packages, create a ~/.pypirc file:

[distutils]
index-servers =
  pypi
  pypitest

[pypi]
repository=https://pypi.python.org/pypi
username=your_username
password=your_password

[pypitest]
repository=https://testpypi.python.org/pypi
username=your_username
password=your_password

Create a .pypirc configuration file (2/2)

Passwords can be omitted in ~/.pypirc and written when uploading.

Warning: in Python 3 % must be escaped in password by doubling them. Info: Spaces don't need to be escaped, there must be not quote.

Create an account

Register on the PyPI Live or Test user registration page.

Register the project and upload the distribution

Register the package project:

python setup.py register -r pypi

Upload the package distribution:

python setup.py sdist upload -r pypi

To make sure the package is set up correctly, package can be upload on PyPI's test server first:

python setup.py register -r pypitest
python setup.py sdist upload -r pypitest

Git tips

If the package code source is hosted on a Git repository:

  • a .gitignore can be added (ex: GitHub Python gitignore to avoid uploading dist/ or build/ folders to the remote Git repository.
  • If the Git remote repository is GitHub git tag 0.1 -m "message" & git push --tags origin master can be used to generate and host package

Warning: To use git tag on GitHub there must be only one tool per repository (no multi-tools repository).

References

Presentation

Presentation:

Author:

  • Alexandre ZANNI