Problem to solve:

I wrote a Python application that is supposed to run on Debian 9/10.
The application uses Python 3.5, but has some newer external dependencies.
The application cannot be dockerized. What is the appropriate way to deploy it to production servers?

Part of the problem is that despite Python’s popularity, there is no official way of doing this.

The solution:

Doing my research, I found that some enterprises are building .deb files to distribute their applications to production. Specifically, I found that Spotify created dh-virtualenv as a tool to simplify the process of creating Debian packages for their python projects.

I always wanted to create a Debian package. So I tried this approach.

I must say, that the documentation to create a .deb package for Python is not the best. You can find tutorials to create packages for compiled languages, but for Python is hard to find.
The best tutorial that I found is the following from Py3g.

The application to be packaged as .deb:

To learn to use dh-virtualenv I decided to package an old project with a certain complexity.
My sudoku solver application is an accumulation of technologies that I wanted to learn at the time. Including Flask, Tkinter, and Requests.

The code is not structured as a package (in the Python sense). It has a requirements file but not a setup.py. It does not have a init.py either.

Step #1: Backup your code

Work on a copy of your source code. At this moment my code look like this.

.
├── README.md
├── requirements.txt
└── src
├── cell.py
├── gui_tkinter.py
├── solver_api.py
├── sudoku.py
└── tests.py

Step #2: Get cookiecutter and the dh-virtualev:

Part of the complexity of building a Debian package is dealing with the boiler-plate of files that you need to create.
You can get an excellent template by using cookiecutter.

sudo apt install cookiecutter
cookiecutter https://github.com/Springerle/dh-virtualenv-mold.git

After launching cookiecutter and answering the questions. My code looks like this:
.
├── debian
│   ├── changelog
│   ├── compat
│   ├── control
│   ├── cookiecutter.json
│   ├── copyright
│   ├── rules
│   ├── source
│   │   ├── format
│   │   └── options
│   ├── sudoku.links
│   ├── sudoku.postinst
│   └── sudoku.triggers
├── README.md
├── requirements.txt
└── src
├── cell.py
├── gui_tkinter.py
├── solver_api.py
├── sudoku.py
└── tests.py

Cookiecutter created the Debian package and a considerable amount of the boilerplate needed to create the Debian package.

(optional) Step #3: Create a safe environment to build your project

You can build the image on your local, however, it does no harm to create a safe environment for building your .deb package. In my case, I struggled to find the adequate packages on my Ubuntu 20.04 machine, so instead, I switched to a Debian docker. (One advantage of using docker is that you would write down the environment you need to build your project, hence you will be already beginning the automatization of a future CICD pipeline.)

After struggling with building the package I came with this rule of thumb: Your building environment should be as close as possible to your deployment environment. If you are going to deploy in Debian 9, build using Debian 9; Debian 10, Debian 10; Ubuntu 18.04, Ubuntu 18.04 etc. Given that building a .deb package is an iterative process, you want to reduce to the minimum the possible sources of problems.

You might also find that a configuration might work if you build for example with Debian 10 but it will not work with Debian 9. So choose well the environment from the beginning.

I would even say that if it’s easy for you, if you are building an application with a graphical interface, maybe try to build in an environment with a graphical interface. These recommendations are not strictly necessary but can help you in your debugging process.

The next step is then to touch and edit the Dockerfile for the building environment (could be in a different folder than your project):

touch Dockerfile
vim Dockerfile

The docker image I build for this purpose has this structure:

FROM debian:buster-slim
USER root
RUN apt-get update && apt-get install -y build-essential debhelper devscripts equivs python3-venv python3-dev dh-virtualenv python3-setuptools

I use Debian 9 because the real project that I have in mind should be deployed to Debian 9. In my case I had to tweak several things to make it work, with Debian 10 it was way easier.

Run you docker and share a volume

docker build -t dh_virtualenv:debian9 ./
docker run -it -v $PWD:/home/ dh_virtualenv:debian9 /bin/bash

I use docker to build the image, but given that I have a graphical part, I will deploy at the end to a VM with desktop to check that everything is working correctly.

Step #4 (iteration #1): Try to build

There are some missing dependencies that you must install that are in the debian/control file

mk-build-deps --install debian/control

Then you can try to do your first build:

dpkg-buildpackage -uc -us –b

No surprise, the building does not work.

/usr/bin/python3: can't open file 'setup.py': [Errno 2] No such file or directory

The problem is that we are packaging a Python package, but in my code, there is neither a package nor a setup.py and both are part of the building process.

Step #5 (iteration #2): Create setup.py and organize my code as a package

First, to make Python recognize my code as a package instead of a bunch of modules, I need to create an empty init.py, I will also rename my new package to “sudoku” instead of src. And finally, I will create a setup.py

mv src sudoku
touch ./sudoku/__init__.py
touch ./setup.py

Here is the content of my first setup.py:

import setuptools

with open("README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="sudoku", # Replace with your own username
    version="0.0.1",
    author="Camilo MATAJIRA",
    author_email="ca.matajira966@gmail.com",
    description="A small example package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/pypa/sampleproject",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    python_requires='>=3.5',
)

Basically, the important line is “packages=setuptools.find_packages()” because it instructs the building mechanism to search for the packages (i.e folders with init.py inside.)

Let’s try to build again. I got this error.

Traceback (most recent call last):
  File "setup.py", line 1, in <module>
    import setuptools
ImportError: No module named setuptools

This error is due to a PATH problem because I know for sure that I installed setuptools in the Dockerfile.

export PYTHONPATH=/usr/lib/python3/dist-packages/

I tried to build, and it seems to work now! It installed Flask (in my requirements.txt) and install the Flask dependencies.
The resulting deb package is one level up in ../

cd ../
dpkg -i ./sudoku_0.1.0_amd64.deb

It installed! To check this, dh_virtualenv will store the packaged application with its virtual environment in /opt/venvs/ . Specifically in /opt/venvs/sudoku/lib/python3.7/site-packages/sudoku/.

Amazing, all my code is there, but how do I launch it? How do I take advantage of the virtual environment?

Step #7 (iteration #4): Add entry_points to setup.py

To create the executable for your project, you need to add the entry_point section to the setup.py. In my case, given that my project has a gui component I will add “gui_scripts”, if instead, it is a console application you should add console_script.

    entry_points={
    "gui_scripts": [
        "sudoku = sudoku.gui_tkinter:main",
    ]
    },

On the left side of the equal sign, I express that I want my executable to be called “sudoku”. On the right-hand side, I tell them where to find this executable, it is in the “sudoku” package, module “gui_tkinter”, function main (it could be any function if you want).

For this, I reorganized the gui_tkinter.py so that It has a method that will launch everything, and I will call it main.

def main():
    gui = SudokuGui()
if __name__ == "__main__":
    main()

To try all these modifications, I will uninstall the package, and built it again, and install it again.

dpkg -r sudoku 
dpkg-buildpackage -uc -us –b
dpkg -i ../sudoku_0.1.0_amd64.deb

Now I find my new sudoku executable inside the virtual environment /opt/venvs/sudoku/bin/sudoku.

I execute it, but now I got import problems.
Traceback (most recent call last):
  File "/opt/venvs/sudoku/bin/sudoku", line 5, in <module>
    from sudoku.gui_tkinter import main
  File "/opt/venvs/sudoku/lib/python3.7/site-packages/sudoku/gui_tkinter.py", line 4, in <module>
    import solver_api
ModuleNotFoundError: No module named 'solver_api'

This has to do with the structure of my code. Despite Python recognize my code as a package I still need to rethink the import statements with my view will be inside a package instead of being random modules lying around.

For example, now that my modules are inside the sudoku package, I should reference them like this:

import solver_api ==> from sudoku import solver_api

Let’s iterate again.

Step #8 (iteration #5): Allow my python program to use system-wide (apt) libraries

Now it launched!!! I see the GUI, but when I try to solve my sudoku puzzle I get

ModuleNotFoundError: no module named 'requests'

I thought that requests was part of the builtins but I was wrong. Probably I had installed the library before on my local and I had forgotten.
To add spicy to this requirement I won’t put requests on the requirements.txt file, Instead, I want to tell dh-virtualenv that I want my virtual environment to work also with the external packages of the system (the ones installed via apt).

To achieve this, we need to make two modifications. The first one involves modifying the debian/contron file to tell it to require the python3-request. I do this by adding python3-request to the Pre-Depends line.

Pre-Depends: dpkg (>= 1.16.1), python3, python3-venv,python3-requests, ${misc:Pre-Depends}

The second modification is to add a flag to dh_virtualenv so that when it creates the virtual environment it also includes the normal packages into the path. For this, we modify the debian/rules file, and add this flag –use-system-packages

DH_VENV_ARGS=--setuptools --builtin-venv --python=$(SNAKE) $(EXTRA_REQUIREMENTS) --use-system-packages \

Now, we built and install again. And voila, a nice error message (the one that we wanted).

dpkg: regarding sudoku_0.1.0_amd64.deb containing sudoku, pre-dependency problem:
 sudoku pre-depends on python3-requests
  python3-requests is not installed.

dpkg: error processing archive sudoku_0.1.0_amd64.deb (--install):
 pre-dependency problem - not installing sudoku
Errors were encountered while processing:
 sudoku_0.1.0_amd64.deb

We then install python3-requests, and the sudoku app is up and running!

/opt/venvs/sudoku/bin/sudoku

Step #8 (iteration #6): Add soft links to /usr/bin

However, no one wants to have their program hidden in such a mysterious path. Dh-virtualenv also has the option to create a soft link in the /usr/bin folder. To create this link we need to modify to debian/.links

opt/venvs/sudoku/bin/sudoku usr/bin/sudoku

dh-virtualenv had already provided the link that I wanted, I just need to uncomment the line.

opt/venvs/sudoku/bin/sudoku usr/bin/sudoku

And now, after building and installing again, just typing sudoku runs my program!

sudoku