Recently I spent an embarrassingly long time trying to figure out how to set up a cron job to run a python script I had recently created.
Before you snicker under your breath there is one caveat you should be aware of. I am a huge fan of Conda and the ability to create different virtual environments for different combinations of python versions and package dependencies. Conda is related to the popular Anaconda python distribution and is very much worth taking the time to understand.
With Conda environments you activate them by typing ‘source activate [Environment Name]’, and deactivate them by typing ‘source deactivate [Environment Name]’. Unfortunately this is not how you set it up to run via cron.
What is needed is to find the path to the python that runs when you are in that particular environment. For me this looks something like:
/home/mypc/miniconda3/envs/Analytics/bin/python
…Where ‘Analytics’ is the name of the virtual environment I created with Conda.
Creating a bash script to run the python script then should look like following.
#!/bin/bash
### Do not do the following it will not work ###
#source activate AVirtualEnvironment
#python mypythonscript.py
#source deactivate AVirtual Environment
### Do this instead ###
/home/mypc/miniconda3/envs/Analytics/bin/python mypythonscript.pyTo get this script to run as a cron task we first need to make sure that this script has permissions to run for all users as follows
chmod a+x runmypythonscript.sh
With this in place we can now go ahead and set up the cron task to run whenever is needed. For more details on how to set up cron jobs have a look at this cron guide from Ubuntu. With any luck this will save you a day or two of debugging and Googling around.
Hey, thanks for posting 🙂 is there any reason why you’re wrapping the commands in a shell script? it should work if you invoke it directly from cron, ie;
* * * * * /home/mypc/miniconda3/envs/Analytics/bin/python mypythonscript.py
Hi Hoolio, you are right that it is possible to directly schedule this in cron. This is just a preference of mine I guess, I like my cron jobs to be a one to one relationship when it comes to projects. For example I often need to run multiple scripts to do a task, this way I can call all of the scripts from within bash script while still having a single entry in cron.
“### Do not do the following it will not work ###”
It works, if you type the full path.
Hi Marquin,
Thank you so much! your post saved possibly days of my time!
Thank you so much for posting this! Huge help.
Hey there!
It seems that this solution doesn’t bring environment variables from conda env into cron’s scope. E.g. in my case /bin/sh doesn’t see GDAL_DATA variable, which is otherwise loaded into scope when doing “source activate xyz”. Anybody found a proper solution yet?
I am also looking for a solution which imports all env variables from conda env.
A huge help! Thanks for sharing this post.
Thanks