env_parallel1
Export environment to gnu parallel
Usage
env_parallel [{--record-env | --session | --end-session}]
[options
for
GNU
Parallel]
Overview
env_parallel is a shell function that exports the current environment to GNU parallel.
If the shell function is not loaded, a dummy script will be run instead that explains how to install the function.
env_parallel is 100 ms slower at startup than pure GNU parallel, and takes up to 30% longer to start a job (typically 15 ms).
Due to the problem with environment space (see below) you are recommended only to transfer the environment that you need.
To help you do that, you can mark names that should not be transferred. This can be done with either --session or --record-env.
# Record the "clean" environment (this only needs to be run once)env_parallel --record-env# Optionally edit ~/.parallel/ignored_vars (only needed once)# Define whatever you want to usemyfunc() { myalias and functions $myvar work. $1.; }alias myalias='echo Aliases'myvar='and variables'# Use --env _ to only transfer the names not in the "empty" environmentenv_parallel --env _ -S localhost myfunc ::: Hooray
Or:
# Do --record-env into $PARALLEL_IGNORED_NAMESenv_parallel --session# Define whatever you want to usemyfunc() { myalias and functions $myvar work. $1.; }alias myalias='echo Aliases'myvar='and variables'# env_parallel will not export names in $PARALLEL_IGNORED_NAMESenv_parallel -S localhost myfunc ::: Hooray# Optionallyenv_parallel --end-session
In csh --session is not supported:
# Record the "clean" environment - this only needs to be run onceenv_parallel --record-env# Optionally edit ~/.parallel/ignored_vars - only needed once# Define whatever you want to usealias myalias 'echo Aliases $myvar \!*.'set myvar='and variables'# Use --env _ to only transfer the names not in the "empty" environmentenv_parallel --env _ -S localhost myalias ::: work
Environment space
By default env_parallel will export all environment variables, arrays, aliases, functions and shell options (see details for the individual shells below).
But this only works if the size of the current environment is smaller than the maximal length of a command and smaller than half of the max if running remotely. E.g. The max size of Bash's command is 128 KB, so env_parallel will fail if 'set | wc -c' is bigger than 128 KB. Technically the limit is in execve(1) which IPC::open3 uses.
Bash completion functions are well-known for taking up well over 128 KB of environment space and the primary reason for causing env_parallel to fail.
Instead you can use --env to specify which variables, arrays, aliases and functions to export as this will only export those with the given name. Or follow the recommended usage in shown in DESCRIPTION.
Options
Same as GNU parallel in addition to these:
-
--end-session -
Undo last
--session
-
--record-env -
Record all names currently defined to be ignored every time running
env_parallelin the future.
-
--session -
Ignore all names currently defined. Aliases, variables, arrays, and functions currently defined will not be transferred.
But names defined after running parallel --session will be transferred.
This is only valid in the running shell, and can be undone with parallel --end-session.
You can run multiple
--sessioninside each other:env_parallel --sessionvar=not# var is transferredenv_parallel -Slocalhost 'echo var is $var' ::: ignoredenv_parallel --session# var is not transferredenv_parallel -Slocalhost 'echo var is $var' ::: ignoredenv_parallel --end-session# var is transferred againenv_parallel -Slocalhost 'echo var is $var' ::: ignored
Supported shells
Ash
Installation
Put this in $HOME/.profile:
. env_parallel.ashE.g. by doing:
echo '. env_parallel.ash' >> $HOME/.profileSupported use
--env is supported to export only the variable, or alias with the given name. Multiple --envs can be given.
--session is supported.
- aliases
-
alias myecho='echo aliases'env_parallel myecho ::: workenv_parallel -S server myecho ::: workenv_parallel --env myecho myecho ::: workenv_parallel --env myecho -S server myecho ::: workalias multiline='echo multilineecho aliases'env_parallel multiline ::: workenv_parallel -S server multiline ::: workenv_parallel --env multiline multiline ::: workenv_parallel --env multiline -S server multiline ::: work
- functions
-
ash cannot list defined functions - thus is not supported.
- variables
-
myvar=variablesenv_parallel echo '$myvar' ::: workenv_parallel -S server echo '$myvar' ::: workenv_parallel --env myvar echo '$myvar' ::: workenv_parallel --env myvar -S server echo '$myvar' ::: work
- arrays
-
Arrays are not supported by Ash.
Bash
Installation
Put this in $HOME/.bashrc:
. env_parallel.bashE.g. by doing:
echo '. env_parallel.bash' >> $HOME/.bashrcSupported use
--env is supported to export only the variable, alias, function, or array with the given name. Multiple --envs can be given.
--session is supported.
- aliases
-
alias myecho='echo aliases'env_parallel myecho ::: workenv_parallel -S server myecho ::: workenv_parallel --env myecho myecho ::: workenv_parallel --env myecho -S server myecho ::: workalias multiline='echo multilineecho aliases'env_parallel 'multiline {};echo but only when followed by a newline' ::: workenv_parallel -S server 'multiline {};echo but only when followed by a newline' ::: workenv_parallel --env multiline 'multiline {};echo but only when followed by a newline' ::: workenv_parallel --env multiline -S server 'multiline {};echo but only when followed by a newline' ::: work
- functions
-
myfunc() { echo functions $*; }env_parallel myfunc ::: workenv_parallel -S server myfunc ::: workenv_parallel --env myfunc myfunc ::: workenv_parallel --env myfunc -S server myfunc ::: work
- variables
-
myvar=variablesenv_parallel echo '$myvar' ::: workenv_parallel -S server echo '$myvar' ::: workenv_parallel --env myvar echo '$myvar' ::: workenv_parallel --env myvar -S server echo '$myvar' ::: work
- arrays
-
myarray=(arrays work, too)env_parallel -k echo '${myarray[{}]}' ::: 0 1 2env_parallel -k -S server echo '${myarray[{}]}' ::: 0 1 2env_parallel -k --env myarray echo '${myarray[{}]}' ::: 0 1 2env_parallel -k --env myarray -S server \echo '${myarray[{}]}' ::: 0 1 2
BUGS
Due to a bug in Bash, aliases containing newlines must be followed by a newline in the command. Some systems are not affected by this bug, but will print a warning anyway.
csh
env_parallel for csh breaks $PARALLEL, so do not use $PARALLEL.
Installation
Put this in $HOME/.cshrc:
source `which env_parallel.csh`E.g. by doing:
echo 'source `which env_parallel.csh`' >> $HOME/.cshrcSupported use
--env is supported to export only the variable, alias, or array with the given name. Multiple --envs can be given.
- aliases
-
alias myecho 'echo aliases'env_parallel myecho ::: workenv_parallel -S server myecho ::: workenv_parallel --env myecho myecho ::: workenv_parallel --env myecho -S server myecho ::: work
- functions
-
Not supported by
csh.
- variables
-
set myvar=variablesenv_parallel echo '$myvar' ::: workenv_parallel -S server echo '$myvar' ::: workenv_parallel --env myvar echo '$myvar' ::: workenv_parallel --env myvar -S server echo '$myvar' ::: work
- arrays with no special chars
-
set myarray=(arrays work, too)env_parallel -k echo \$'{myarray[{}]}' ::: 1 2 3env_parallel -k -S server echo \$'{myarray[{}]}' ::: 1 2 3env_parallel -k --env myarray echo \$'{myarray[{}]}' ::: 1 2 3env_parallel -k --env myarray -S server \echo \$'{myarray[{}]}' ::: 1 2 3
Dash
Installation
Put this in $HOME/.profile:
. env_parallel.dashE.g. by doing:
echo '. env_parallel.dash' >> $HOME/.profileSupported use
--env is supported to export only the variable, or alias with the given name. Multiple --envs can be given.
--session is supported.
- aliases
-
alias myecho='echo aliases'env_parallel myecho ::: workenv_parallel -S server myecho ::: workenv_parallel --env myecho myecho ::: workenv_parallel --env myecho -S server myecho ::: workalias multiline='echo multilineecho aliases'env_parallel multiline ::: workenv_parallel -S server multiline ::: workenv_parallel --env multiline multiline ::: workenv_parallel --env multiline -S server multiline ::: work
- functions
-
dash cannot list defined functions - thus is not supported.
- variables
-
myvar=variablesenv_parallel echo '$myvar' ::: workenv_parallel -S server echo '$myvar' ::: workenv_parallel --env myvar echo '$myvar' ::: workenv_parallel --env myvar -S server echo '$myvar' ::: work
- arrays
-
dash does not support arrays.
fish
Installation
Put this in $HOME/.config/fish/config.fish:
source (which env_parallel.fish)E.g. by doing:
echo 'source (which env_parallel.fish)' \>> $HOME/.config/fish/config.fish
Supported use
--env is supported to export only the variable, alias, function, or array with the given name. Multiple --envs can be given.
--session is supported.
- aliases
-
alias myecho 'echo aliases'env_parallel myecho ::: workenv_parallel -S server myecho ::: workenv_parallel --env myecho myecho ::: workenv_parallel --env myecho -S server myecho ::: work
- functions
-
function myfuncecho functions $argvendenv_parallel myfunc ::: workenv_parallel -S server myfunc ::: workenv_parallel --env myfunc myfunc ::: workenv_parallel --env myfunc -S server myfunc ::: work
- variables
-
set myvar variablesenv_parallel echo '$myvar' ::: workenv_parallel -S server echo '$myvar' ::: workenv_parallel --env myvar echo '$myvar' ::: workenv_parallel --env myvar -S server echo '$myvar' ::: work
- arrays
-
Arrays normally work, but fail intermitently.
set myarray arrays work, tooenv_parallel -k echo '$myarray[{}]' ::: 1 2 3env_parallel -k -S server echo '$myarray[{}]' ::: 1 2 3env_parallel -k --env myarray echo '$myarray[{}]' ::: 1 2 3env_parallel -k --env myarray -S server \echo '$myarray[{}]' ::: 1 2 3
ksh
Installation
Put this in $HOME/.kshrc:
source env_parallel.kshE.g. by doing:
echo 'source env_parallel.ksh' >> $HOME/.kshrcSupported use
--env is supported to export only the variable, alias, function, or array with the given name. Multiple --envs can be given.
--session is supported.
- aliases
-
alias myecho='echo aliases'env_parallel myecho ::: workenv_parallel -S server myecho ::: workenv_parallel --env myecho myecho ::: workenv_parallel --env myecho -S server myecho ::: workalias multiline='echo multilineecho aliases'env_parallel multiline ::: workenv_parallel -S server multiline ::: workenv_parallel --env multiline multiline ::: workenv_parallel --env multiline -S server multiline ::: work
- functions
-
myfunc() { echo functions $*; }env_parallel myfunc ::: workenv_parallel -S server myfunc ::: workenv_parallel --env myfunc myfunc ::: workenv_parallel --env myfunc -S server myfunc ::: work
- variables
-
myvar=variablesenv_parallel echo '$myvar' ::: workenv_parallel -S server echo '$myvar' ::: workenv_parallel --env myvar echo '$myvar' ::: workenv_parallel --env myvar -S server echo '$myvar' ::: work
- arrays
-
myarray=(arrays work, too)env_parallel -k echo '${myarray[{}]}' ::: 0 1 2env_parallel -k -S server echo '${myarray[{}]}' ::: 0 1 2env_parallel -k --env myarray echo '${myarray[{}]}' ::: 0 1 2env_parallel -k --env myarray -S server \echo '${myarray[{}]}' ::: 0 1 2
mksh
Installation
Put this in $HOME/.mkshrc:
source env_parallel.mkshE.g. by doing:
echo 'source env_parallel.mksh' >> $HOME/.mkshrcSupported use
--env is supported to export only the variable, alias, function, or array with the given name. Multiple --envs can be given.
--session is supported.
- aliases
-
alias myecho='echo aliases'env_parallel myecho ::: workenv_parallel -S server myecho ::: workenv_parallel --env myecho myecho ::: workenv_parallel --env myecho -S server myecho ::: workalias multiline='echo multilineecho aliases'env_parallel multiline ::: workenv_parallel -S server multiline ::: workenv_parallel --env multiline multiline ::: workenv_parallel --env multiline -S server multiline ::: work
- functions
-
myfunc() { echo functions $*; }env_parallel myfunc ::: workenv_parallel -S server myfunc ::: workenv_parallel --env myfunc myfunc ::: workenv_parallel --env myfunc -S server myfunc ::: work
- variables
-
myvar=variablesenv_parallel echo '$myvar' ::: workenv_parallel -S server echo '$myvar' ::: workenv_parallel --env myvar echo '$myvar' ::: workenv_parallel --env myvar -S server echo '$myvar' ::: work
- arrays
-
myarray=(arrays work, too)env_parallel -k echo '${myarray[{}]}' ::: 0 1 2env_parallel -k -S server echo '${myarray[{}]}' ::: 0 1 2env_parallel -k --env myarray echo '${myarray[{}]}' ::: 0 1 2env_parallel -k --env myarray -S server \echo '${myarray[{}]}' ::: 0 1 2
pdksh
Installation
Put this in $HOME/.profile:
source env_parallel.pdkshE.g. by doing:
echo 'source env_parallel.pdksh' >> $HOME/.profileSupported use
--env is supported to export only the variable, alias, function, or array with the given name. Multiple --envs can be given.
--session is supported.
- aliases
-
alias myecho="echo aliases";env_parallel myecho ::: work;env_parallel -S server myecho ::: work;env_parallel --env myecho myecho ::: work;env_parallel --env myecho -S server myecho ::: work
- functions
-
myfunc() { echo functions $*; };env_parallel myfunc ::: work;env_parallel -S server myfunc ::: work;env_parallel --env myfunc myfunc ::: work;env_parallel --env myfunc -S server myfunc ::: work
- variables
-
myvar=variables;env_parallel echo "\$myvar" ::: work;env_parallel -S server echo "\$myvar" ::: work;env_parallel --env myvar echo "\$myvar" ::: work;env_parallel --env myvar -S server echo "\$myvar" ::: work
- arrays
-
myarray=(arrays work, too);env_parallel -k echo "\${myarray[{}]}" ::: 0 1 2;env_parallel -k -S server echo "\${myarray[{}]}" ::: 0 1 2;env_parallel -k --env myarray echo "\${myarray[{}]}" ::: 0 1 2;env_parallel -k --env myarray -S server \echo "\${myarray[{}]}" ::: 0 1 2
sh
Installation
Put this in $HOME/.profile:
. env_parallel.shE.g. by doing:
echo '. env_parallel.sh' >> $HOME/.profileSupported use
--env is supported to export only the variable, or alias with the given name. Multiple --envs can be given.
--session is supported.
- aliases
-
sh does not support aliases.
- functions
-
myfunc() { echo functions $*; }env_parallel myfunc ::: workenv_parallel -S server myfunc ::: workenv_parallel --env myfunc myfunc ::: workenv_parallel --env myfunc -S server myfunc ::: work
- variables
-
myvar=variablesenv_parallel echo '$myvar' ::: workenv_parallel -S server echo '$myvar' ::: workenv_parallel --env myvar echo '$myvar' ::: workenv_parallel --env myvar -S server echo '$myvar' ::: work
- arrays
-
sh does not support arrays.
tcsh
env_parallel for tcsh breaks $PARALLEL, so do not use $PARALLEL.
Installation
Put this in $HOME/.tcshrc:
source `which env_parallel.tcsh`E.g. by doing:
echo 'source `which env_parallel.tcsh`' >> $HOME/.tcshrcSupported use
--env is supported to export only the variable, alias, or array with the given name. Multiple --envs can be given.
- aliases
-
alias myecho 'echo aliases'env_parallel myecho ::: workenv_parallel -S server myecho ::: workenv_parallel --env myecho myecho ::: workenv_parallel --env myecho -S server myecho ::: work
- functions
-
Not supported by
tcsh.
- variables
-
set myvar=variablesenv_parallel echo '$myvar' ::: workenv_parallel -S server echo '$myvar' ::: workenv_parallel --env myvar echo '$myvar' ::: workenv_parallel --env myvar -S server echo '$myvar' ::: work
- arrays with no special chars
-
set myarray=(arrays work, too)env_parallel -k echo \$'{myarray[{}]}' ::: 1 2 3env_parallel -k -S server echo \$'{myarray[{}]}' ::: 1 2 3env_parallel -k --env myarray echo \$'{myarray[{}]}' ::: 1 2 3env_parallel -k --env myarray -S server \echo \$'{myarray[{}]}' ::: 1 2 3
Zsh
Installation
Put this in $HOME/.zshrc:
. env_parallel.zshE.g. by doing:
echo '. env_parallel.zsh' >> $HOME/.zshenvSupported use
--env is supported to export only the variable, alias, function, or array with the given name. Multiple --envs can be given.
--session is supported.
- aliases
-
alias myecho='echo aliases'env_parallel myecho ::: workenv_parallel -S server myecho ::: workenv_parallel --env myecho myecho ::: workenv_parallel --env myecho -S server myecho ::: workalias multiline='echo multilineecho aliases'env_parallel multiline ::: workenv_parallel -S server multiline ::: workenv_parallel --env multiline multiline ::: workenv_parallel --env multiline -S server multiline ::: work
- functions
-
myfunc() { echo functions $*; }env_parallel myfunc ::: workenv_parallel -S server myfunc ::: workenv_parallel --env myfunc myfunc ::: workenv_parallel --env myfunc -S server myfunc ::: work
- variables
-
myvar=variablesenv_parallel echo '$myvar' ::: workenv_parallel -S server echo '$myvar' ::: workenv_parallel --env myvar echo '$myvar' ::: workenv_parallel --env myvar -S server echo '$myvar' ::: work
- arrays
-
myarray=(arrays work, too)env_parallel -k echo '${myarray[{}]}' ::: 1 2 3env_parallel -k -S server echo '${myarray[{}]}' ::: 1 2 3env_parallel -k --env myarray echo '${myarray[{}]}' ::: 1 2 3env_parallel -k --env myarray -S server \echo '${myarray[{}]}' ::: 1 2 3
Exit status
Same as GNU parallel.
Dependencies
env_parallel uses GNU parallel.
Releases
6 releases since Debian 9 (Stretch); the text changed 4 times.
--end-session since Debian 12 (Bookworm);
--record-env since Debian 12 (Bookworm);
--session since Debian 12 (Bookworm).