Yup, definitely use virtual environments. They contain everything (Python and libs) in one folder, which you can always delete to just start over again without affecting anything on your system.
Python
May 19, 2020
I know you'll hate me for saying this, but Python is a mess. I love programming in Python, but holy crap-a-reno is dealing with all the modules a pain-in-the-ass-a-who. Then python 3 came out. Now I have a complete mess of modules on my machine. I have some modules that are system wide and others that are install just for me. I have no idea how it got so screwed up. It's probably that my Mac is 6 years old and it's just been a slow module death. I'd love to wipe it all clean and start over. I think that would just make it worse.
I'm sure there is some logic behind it, and I'm sure all the python pros are laughing, but getting python working for the average user is nuts. I'm trying to walk some people through it now and it's just impossible.
Jaap
May 19, 2020
Little Cat
May 20, 2020
Yeah I agree, keeping track of what modules you have install system-wide can be a mess, especially since there seems to be no way of doing so (believe me, I have lots and lots and LOTS of modules installed in my system).
Like the other comments pointed out, use a virtual environment for this kind of situation. Dylan's comment seems to explained it all out for you.
Like the other comments pointed out, use a virtual environment for this kind of situation. Dylan's comment seems to explained it all out for you.
Winfried Maus
May 20, 2020
I'm with you. Deploying Python code is a total disaster, it's among the messiest things out there. There should be a simple packing tool that "automagically" takes care of that for you, but there just isn't one. (Emphasis on SIMPLE.) This situation really kills the whole "Python is awesome" myth for me.
Jaap
May 20, 2020
Oh yeah and using pipenv (instead of pip) can help too. It e.g. has: version locking and also has a nice way to track (sub)dependancies with "pipenv graph". Also it has a nice way to create/handle virtual environments.
Stephen
May 20, 2020
Came here to say to use virtualenv and I see its definitely already been mentioned. It's SO helpful in not messing up your machine when it comes to installing modules. I think the Mac is a little messy to begin with because of how OS X has its own python version, and you can also use your own. Then you have to remember which python is the one you have. It is indeed messy. I develop in Python every day and there are some pretty great things about it, but dependency management is kind of an art form.
Happy hacking!
Happy hacking!
Ron Gilbert
May 20, 2020
The real problem here is I need to have other people install python and run my python scripts. The amount of technical knowledge varies. They are all comfortable with the command line, but they could be on a Mac or Windows. It's been a complete nightmare with Python. I need something where I can tell them "run this..." and it's all installed. I'll look into virtual environments.
Jaap
May 20, 2020
In that case, don't fully remember, but I think a colleague of mine tried this and was pretty happy about it. He also needed to package his Python app for colleagues that were not great with command line (who needed to install & maintain it on hardware used by all kind of visiting users) :
https://www.pyinstaller.org
https://www.pyinstaller.org
Joel
May 20, 2020
I fully agree that Python is incredibly messy, despite being a very useful tool. You probably already know about it, but Anaconda is a very nice way of at least cleaning up the install process.
Matej
May 20, 2020
Like others have pointed out, virtual environments are the only way to go. We use Python a lot professionally, and we ask other people to run our code all the time. So all our projects' instructions start with the boilerplate of python3 -m venv .venv ; . .venv/bin/activate ; pip install -r requirements.txt. This then works on mac, linux, windows, anywhere.
Ron Gilbert
May 20, 2020
(repeatably bangs head on desk)
Didn't it occur to anyone that if you need tools like this, your fundamental project is broken? I guess it's just too large to stop at this point.
I looked into pyinstaller and it broken horribly on Mac Catalina.
I'll look at virtual environments next. Wish me luck.
Didn't it occur to anyone that if you need tools like this, your fundamental project is broken? I guess it's just too large to stop at this point.
I looked into pyinstaller and it broken horribly on Mac Catalina.
I'll look at virtual environments next. Wish me luck.
Jaap
May 20, 2020
Definitely aware that a thing like pyinstaller is just a patchwork tool to work around the shortcomings...
Actually for passing around easy to run executables, I would never use Python.
Probably the fastest solution would be to switch language, heh.
(using Python for web development mostly, where this is not an issue)
Actually for passing around easy to run executables, I would never use Python.
Probably the fastest solution would be to switch language, heh.
(using Python for web development mostly, where this is not an issue)
Ron Gilbert
May 20, 2020
I thought about writing all my quick tools in C, but then I have the issue of compiling for a platform. I guess I'm just disappointed that something like python doesn't solve all this. Also, a major library I use (and have no desire to rewrite) is all in python. Python works great when it's just me.
Jaap
May 20, 2020
appreciate that view on Python.
Well I guess then the best approach is indeed the virtual env stuff and what Matej said should be a possible one-liner to go (after Python3.x + pip/pipenv was installed on whoever needs to run it).
Curious to hear what solution will work out eventually
Well I guess then the best approach is indeed the virtual env stuff and what Matej said should be a possible one-liner to go (after Python3.x + pip/pipenv was installed on whoever needs to run it).
Curious to hear what solution will work out eventually
Phil
May 21, 2020
My advice: never touch the system python.
Install the exact version you need using pyenv (https://github.com/pyenv/pyenv) and manage all your project in virtual envs.
I found as well pipx incredibly useful to install python tools (they get installed in their own virtual envs with their dependencies).
Since I'm using these techniques I've not anymore any problems with modules or conflicting dependencies ...
Cheers
Install the exact version you need using pyenv (https://github.com/pyenv/pyenv) and manage all your project in virtual envs.
I found as well pipx incredibly useful to install python tools (they get installed in their own virtual envs with their dependencies).
Since I'm using these techniques I've not anymore any problems with modules or conflicting dependencies ...
Cheers
DerSebi
May 21, 2020
I've had some good success with Nuitka (https://nuitka.net/) just compiling to a native executable. For the few cases where I had to distribute Python Tools it just worked (TM).
Bo J.
May 21, 2020
Have ye a shrubbery?
Ferdi van de Kamp
May 21, 2020
You may want to look at poetry. [link] (https://python-poetry.org/) and give your stuff a test run using a docker image to absolutely make doubly sure it doesn't only work on your personal computer.
M. Web
May 25, 2020
You are sadly not the first to encounter this. https://xkcd.com/1987/ This is exactly the reason why I reduced the number of scripts I write in Python today. I only use it for personal scripts and when it starts to grow or if I need to distribute it I usually switch to Go.
But virtual env is the way to go this should work on Mac and Windows.
But virtual env is the way to go this should work on Mac and Windows.
Francisco Guimaraes
May 26, 2020
Call me old fashioned, but I remmber when you had all your dependencies in a "lib" folder and that was that. No need for installing or checking cross dependencies or --fix-missing or any of that
JP
May 26, 2020
I love Python and this is totally fair. macOS is Python on Hard Mode because they include an ancient version of it and refuse to update it. I use PyInstaller to distribute Playscii but yeah I'm sure it doesn't play well with OS versions that get really intense and locking down everything.
Eliug
May 26, 2020
The virtualenvs + requirements.txt is the way to go. If you require another dependency, do a "pip freeze > requirements.txt" after the install. Then pass this file with the rest of your source code in the VCS.
When someone else want to use your code, he simply create a virtualenv and then run "pip install -r requirements.txt" and voila! there is a virtualenv with the exact version than yours!. These commands wokr across operating systems (it's my setup at work with a mixed environment of windows, mac and linux).
I hope you don't drop the towel, python is a wonderful environment for programming!
When someone else want to use your code, he simply create a virtualenv and then run "pip install -r requirements.txt" and voila! there is a virtualenv with the exact version than yours!. These commands wokr across operating systems (it's my setup at work with a mixed environment of windows, mac and linux).
I hope you don't drop the towel, python is a wonderful environment for programming!
Pierre
May 26, 2020
Regarding the body of your article, and the fact that your system is a mess right now, it's probably because at some point you installed Python modules using different methods, so the modules ended up being installed differently on your system, and that's why you're running into all these problems. As mentioned by others above, the solution to this are virtual environments:
# cd into your project directory, then create a virtual env called `venv`:
$ python3 -m venv venv
# activate the virtual environment:
$ . venv/bin/activate
# from now on, you are in the venv as shown by the (venv) prefix in your command line.
# you can use pip to install any module you need:
(venv) $ pip install requests
...
# when you want to leave your virtual environment:
(venv) $ deactivate
$
Some other thoughts:
- Don't use Python 2.x (it's end of life since January), only use Python 3.x
- Never run `sudo pip install ...`, that would install Python modules in the system and might clash with your OS package manager
Regarding the distribution of your tools, it is indeed a really big pain point, especially in a cross-platform World (if you were just targetting Linux, you could use some packaging method such as Snap by Canonical). If your tool is comprised of several modules, but only uses the standard library, Python can create a zip archive that can be run as a Python program (<https://docs.python.org/3/library/zipapp.html>;). Otherwise, I heard good things about Nuitka (mentioned above).
Good luck!
# cd into your project directory, then create a virtual env called `venv`:
$ python3 -m venv venv
# activate the virtual environment:
$ . venv/bin/activate
# from now on, you are in the venv as shown by the (venv) prefix in your command line.
# you can use pip to install any module you need:
(venv) $ pip install requests
...
# when you want to leave your virtual environment:
(venv) $ deactivate
$
Some other thoughts:
- Don't use Python 2.x (it's end of life since January), only use Python 3.x
- Never run `sudo pip install ...`, that would install Python modules in the system and might clash with your OS package manager
Regarding the distribution of your tools, it is indeed a really big pain point, especially in a cross-platform World (if you were just targetting Linux, you could use some packaging method such as Snap by Canonical). If your tool is comprised of several modules, but only uses the standard library, Python can create a zip archive that can be run as a Python program (<https://docs.python.org/3/library/zipapp.html>;). Otherwise, I heard good things about Nuitka (mentioned above).
Good luck!
Tramb
May 26, 2020
Oh yeah deploying apps in Python is a mess, don't let people try to make you believe otherwise.
I switched all of my scripts to Racket, and I just compile those trivially.
No more pip/apt/package manager of the day/venv bullshit and a clean expressive language that didn't reinvent the wheel and even allow you to compute stuff with good performance when needed (CPython is a joke).
Basically, compiling apps is not really a problem when you're out of the C or C++ world. Racket, Rust, or Go have streamlined way to do so.
I switched all of my scripts to Racket, and I just compile those trivially.
No more pip/apt/package manager of the day/venv bullshit and a clean expressive language that didn't reinvent the wheel and even allow you to compute stuff with good performance when needed (CPython is a joke).
Basically, compiling apps is not really a problem when you're out of the C or C++ world. Racket, Rust, or Go have streamlined way to do so.
Kostas Bampionitakis
May 27, 2020
I am a Python developer on a Mac and i can tell you virtualenv is a life saver on that matter. I would also suggest to train yourself on docker development. That's the ultimate development environment and you can keep a clean environment for all your projects.
Mr. Pinsky
May 28, 2020
Well, software dependencies are a tough problem. The best approach is clearly to have a rigid and robust package management at the OS level. Many Linux systems have had this for many years. The point is, packaging the software should be done by the OS people and not by app or module developers. Only the former truly understand how to avoid conflicts and dependency problems. On a system like Debian or Ubuntu, you will find a variety of Python modules prepackaged and ready to install, so it is a one-liner to install what you need, unless you have fairly exotic needs or need bleeding-edge tech. And thanks to the package manager, it is super easy to add and remove modules while always having a 'sane' setup. It is also easy and reliable to reproduce a certain setup on a new computer.
Nils Breunese
May 29, 2020
Mr. Pinsky, using the OS package manager only works if you have a single OS/distribution in your development environment (Ron mentioned Mac, Windows and Linux) and if this package manager offers all the libs you need, with exactly the versions you need, which may or may not be the case. Those packages are fine as dependencies for software offered by that particular package manager, but probably not suitable for development of your multi-platform game. So no, I wouldn't go that route, but use virtual envs instead. Or maybe go even one step further and have your build create a Docker image, so you really know that everyone working on the project will run the exact same thing.
Basto
Jun 13, 2020
I second the recommendation for Anaconda. We are distributing some scientific software with a lot of nightmarish dependencies on Windows, Linux, Mac. (Catalina also broke our software, but Anaconda itself got catalina compatible, so we just needed to repack the installer).
Similar to Windows, Python on OSX has no "standard" install. Many people on OSX use homebrew (the right way for a system wide python), scientists use anaconda; then there are some legacy python installs here and there. Before you start setting up your next try, you should get a clean slate.
If you can guarantee that your customers / coworkers always have an internet connection, you can just distribute a environment.yml file and everybody can sync their environment. Only, if you need explicit offline installers, you will run into problems sometimes. The other thing Anaconda can do for you is to synchronize your build environment, by getting everybody the same GCC + sysroot.
Similar to Windows, Python on OSX has no "standard" install. Many people on OSX use homebrew (the right way for a system wide python), scientists use anaconda; then there are some legacy python installs here and there. Before you start setting up your next try, you should get a clean slate.
If you can guarantee that your customers / coworkers always have an internet connection, you can just distribute a environment.yml file and everybody can sync their environment. Only, if you need explicit offline installers, you will run into problems sometimes. The other thing Anaconda can do for you is to synchronize your build environment, by getting everybody the same GCC + sysroot.
Kisai
Jul 29, 2020
I'm actually of the opinion that we need to go backwards for gamedev. All these "too many cooks in the kitchen" scripting languages have resulted in the scripting languages being pulled in too many directions that make them too hard to use as a game scripting language, too dependency dependent to use as a generic console scripting language, and too "trying to be too much like C++" as all scripting languages eventually start looking like.
If I wanted C++ I would have just coded a game directly in C++. Anyway, we might be justified just compressing, encrypting and placing the assets directly into C header files, which then when the linker runs it automatically strips all the assets not actually referenced in the game. Then just using a much smaller game-specific/developer-specific scripting language to deal with the game's localization.
If I wanted C++ I would have just coded a game directly in C++. Anyway, we might be justified just compressing, encrypting and placing the assets directly into C header files, which then when the linker runs it automatically strips all the assets not actually referenced in the game. Then just using a much smaller game-specific/developer-specific scripting language to deal with the game's localization.
Add your comment:
But here's the trick: use virtual environments religiously. Make a virtual environment for every project - ideally inside that project's source directory, or wherever you're deploying it. This isn't immediately obvious coming from Python 2 since it used to be kind of a third party thing with a couple different approaches, but with Python 3 it's easy and it works out of the box: `python3 -m venv .venv` creates a virtual environment in a directory named ".venv" (I am of course subtly trying to trick you into doing it my way :b). This environment has plain old Python and not much else. Now you can pip install stuff to there like `./.venv/bin/pip install my-package`, and run stuff inside the environment like `./.venv/bin/python3 example.py` or `./.venv/bin/some-launcher-script`. If your own project is using setuptools and everything it can work quite smoothly! (And then you realize setuptools solves about half of the problems you need it to solve if you haven't dutifully broken the non-Python parts of your project into their own build system and everything gets gross again, but it's good for a while).
Anyway, make a bunch of virtual environments and you can forget the system and user Python environments ever existed, I promise :)