Oh, I'm with you on this one! It gets to be completely insane. And then you have to _distribute_ this stuff and it turns out you're better off bundling all the dependencies yourself.

But here's the trick: use virtual environments religiously. Make a virtual environment for every project - ideally inside that project's source directory, or wherever you're deploying it. This isn't immediately obvious coming from Python 2 since it used to be kind of a third party thing with a couple different approaches, but with Python 3 it's easy and it works out of the box: `python3 -m venv .venv` creates a virtual environment in a directory named ".venv" (I am of course subtly trying to trick you into doing it my way :b). This environment has plain old Python and not much else. Now you can pip install stuff to there like `./.venv/bin/pip install my-package`, and run stuff inside the environment like `./.venv/bin/python3 example.py` or `./.venv/bin/some-launcher-script`. If your own project is using setuptools and everything it can work quite smoothly!  (And then you realize setuptools solves about half of the problems you need it to solve if you haven't dutifully broken the non-Python parts of your project into their own build system and everything gets gross again, but it's good for a while).

Anyway, make a bunch of virtual environments and you can forget the system and user Python environments ever existed, I promise :)

Yeah I agree, keeping track of what modules you have install system-wide can be a mess, especially since there seems to be no way of doing so (believe me, I have lots and lots and LOTS of modules installed in my system).

Like the other comments pointed out, use a virtual environment for this kind of situation.  Dylan's comment seems to explained it all out for you.

Oh yeah and using pipenv (instead of pip) can help too. It e.g. has: version locking and also has a nice way to track (sub)dependancies with "pipenv graph". Also it has a nice way to create/handle virtual environments.

The real problem here is I need to have other people install python and run my python scripts.  The amount of technical knowledge varies. They are all comfortable with the command line, but they could be on a Mac or Windows.  It's been a complete nightmare with Python.  I need something where I can tell them "run this..." and it's all installed.  I'll look into virtual environments.

I fully agree that Python is incredibly messy, despite being a very useful tool. You probably already know about it, but Anaconda is a very nice way of at least cleaning up the install process.

(repeatably bangs head on desk)

Didn't it occur to anyone that if you need tools like this, your fundamental project is broken?  I guess it's just too large to stop at this point.

I looked into pyinstaller and it broken horribly on Mac Catalina.

I'll look at virtual environments next.  Wish me luck.

I thought about writing all my quick tools in C, but then I have the issue of compiling for a platform.  I guess I'm just disappointed that something like python doesn't solve all this.  Also, a major library I use (and have no desire to rewrite) is all in python.  Python works great when it's just me.

My advice: never touch the system python.
Install the exact version you need using pyenv (https://github.com/pyenv/pyenv) and manage all your project in virtual envs.

I found as well pipx incredibly useful to install python tools (they get installed in their own virtual envs with their dependencies).

Since I'm using these techniques I've not anymore any problems with modules or conflicting dependencies ...

Cheers

Have ye a shrubbery?

You are sadly not the first to encounter this.  https://xkcd.com/1987/ This is exactly the reason why I reduced the number of scripts I write in Python today. I only use it for personal scripts and when it starts to grow or if I need to distribute it I usually switch to Go.
But virtual env is the way to go this should work on Mac and Windows.

I love Python and this is totally fair. macOS is Python on Hard Mode because they include an ancient version of it and refuse to update it. I use PyInstaller to distribute Playscii but yeah I'm sure it doesn't play well with OS versions that get really intense and locking down everything.

Regarding the body of your article, and the fact that your system is a mess right now, it's probably because at some point you installed Python modules using different methods, so the modules ended up being installed differently on your system, and that's why you're running into all these problems. As mentioned by others above, the solution to this are virtual environments:

# cd into your project directory, then create a virtual env called `venv`:
$ python3 -m venv venv
# activate the virtual environment:
$ . venv/bin/activate
# from now on, you are in the venv as shown by the (venv) prefix in your command line.
# you can use pip to install any module you need:
(venv) $ pip install requests
...
# when you want to leave your virtual environment:
(venv) $ deactivate
$

Some other thoughts:

- Don't use Python 2.x (it's end of life since January), only use Python 3.x
- Never run `sudo pip install ...`, that would install Python modules in the system and might clash with your OS package manager

Regarding the distribution of your tools, it is indeed a really big pain point, especially in a cross-platform World (if you were just targetting Linux, you could use some packaging method such as Snap by Canonical). If your tool is comprised of several modules, but only uses the standard library, Python can create a zip archive that can be run as a Python program (<https://docs.python.org/3/library/zipapp.html>;). Otherwise, I heard good things about Nuitka (mentioned above).

Good luck!

I am a Python developer on a Mac and i can tell you virtualenv is a life saver on that matter. I would also suggest to train yourself on docker development. That's the ultimate development environment and you can keep a clean environment for all your projects.

Mr. Pinsky, using the OS package manager only works if you have a single OS/distribution in your development environment (Ron mentioned Mac, Windows and Linux) and if this package manager offers all the libs you need, with exactly the versions you need, which may or may not be the case. Those packages are fine as dependencies for software offered by that particular package manager, but probably not suitable for development of your multi-platform game. So no, I wouldn't go that route, but use virtual envs instead. Or maybe go even one step further and have your build create a Docker image, so you really know that everyone working on the project will run the exact same thing.