Skip to main content

Launch

Launch is a utility command meant to facilitate locally launching and shutting down an entire Cogment project at once.

When launch is used, a set of processes will be launched to run in parallel, as described by a YAML definition file. Once any of these processes terminates, all other ones will be terminated as well.

The order of of the script execution is undefined, which is to say that they may not start in the order they are defined, and may not always start in the same order. Script dependency can be used if a certain order must be used.

Command line​

cogment launch [options] filename [args...]

filename: Name (and path) of the YAML definition file describing the processes to launch.

options:

[-q], [--quiet]: Disable some of the output generated by the launcher. Process output is never disabled by this option. To increase the level, more "q" can be added, up to -qqq disabling all launcher generated output except errors.

args: These are arguments that will be used in variable substitution in the definition file. If some arguments start with a dash (-) a double dash (--) must be used to separate these from the launch options.

E.g.:

$ cogment launch --quiet ./launch.yaml 1000 8
$ cogment launch -qq ./launch.yaml 1000 8
$ cogment launch --quiet ./launch.yaml -- --file ./myfile --match_all

definition file​

The launch definition file is a YAML formatted file where the details of the parallel processes to run are defined. The file consists of at least a scripts top level node, and may also contain a global top level node.

Nodes​

Scripts​

The scripts node contains the details of the processes to run. Each node under scripts represents a process to run. The name of the node becomes the identity of the script/process and serves to identify the process output.

Each process will run the contents of the commands list in sequence (i.e. the next command will start when the previous ends). Each command runs in an independent environment (e.g. environment variables set by one command will not be seen by the others).

E.g.:

scripts:
    process_a: # The name of this process is "process_a"
        commands:
            - ["retrieve_db.sh"]
            - ["python3", "env/main.py"]

    process_b: # The name of this process is "process_b"
        commands:
            - ["cogment", "service", "orchestrator"]

The output will then look something like this:

2023-06-30T22:24:08Z [TRACE ] [process_a] Ready
2023-06-30T22:24:08Z [TRACE ] [process_a:(1/2)] Launch [retrieve_db.sh]
2023-06-30T22:24:08Z [TRACE ] [process_b] Ready
2023-06-30T22:24:08Z [stdout] [process_a:(1/2)] Retrieving default database...
2023-06-30T22:24:08Z [TRACE ] [process_b:(1/1)] Launch [cogment services orchestrator]
2023-06-30T22:24:08Z [stderr] [process_b:(1/1)] 2023-06-30T22:24:08Z [INFO] [cmd] starting the orchestrator service [version:2.16.0]
2023-06-30T22:24:09Z [stderr] [process_a:(1/2)] Record #2278 inconsistent
2023-06-30T22:24:12Z [stdout] [process_a:(1/2)] Database retrieved in /app/sb/
2023-06-30T22:24:12Z [TRACE ] [process_a:(1/2)] Completed
2023-06-30T22:24:12Z [TRACE ] [process_a:(2/2)] Launch [python3 env/main.py]
...

Notes:

  • "[TRACE ]" is low level information from the launcher (it could also be "[INFO ]" for more important information).
  • "[stdout]", and "[stderr]" are the output from the process.
  • "process_a" and "process_b" are the names given to the processes in the definition file.
  • "(1/2)" means that it is the first command out of two for that process.
  • The times/dates are in RFC3339 format.
Environment Variables​

You can set environment variables using the environment node of the process. These will be part of the environment of all commands in the process.

E.g:

scripts:
    orchestrator:
        environment:
            COGMENT_ORCHESTRATOR_ACTOR_PORT: 9000
            COGMENT_LIFECYCLE_PORT: 9000
        commands:
            - ["cogment", "services", "orchestrator"]
Working folder​

By default, the current working folder is set to the folder containing the launch definition file (not the folder where the Launch command is executed). You can change the working folder with the folder node for each process.

E.g.:

# Definition file is in "/home/user"
scripts:
    actor_alpha:
        folder: ./actors/alpha # Working folder is "/home/user/actors/alpha"
        commands:
            - ["python3", "main.py"]
Quiet​

You can control the process output by setting this value to True or False. By default this is False, in which case standard process output (stdout and stderr) are also output by the launcher. If set to True, standard process output is discarded and not output by the launcher.

E.g.:

scripts:
    setup:
        quiet: True
        commands:
            - ["python3", "setup.py"]
Dependency (Cogment >= 2.16)​

Inter-process dependency can also be defined using the depends_on and ready_output nodes of the script.

The depends_on node is a list of process names that this script depends on. Which means that all the listed processes must be "ready" before the current process is started.

The ready_output node is a regex (regular expression) string to search for in the output of the process. When the regex matches the output, the process is considered to be "ready". If no regex string is provided (or the regex string is empty), the process is considered "ready" as soon as it starts.

E.g.:

scripts:
    process_a:
        quiet: True
        commands:
            - ["retrieve_db.sh"]
            - ["python3", "env/main.py"]
        ready_output: '^Database retrieved in'

    process_b:
        depends_on:
            - process_a
        commands:
            - ["cogment", "service", "orchestrator"]

Notes:

  • It is better to use single quotes to define the regex string to prevent YAML from interpreting control characters (e.g. backslash).
  • Be aware that colors and other terminal controls (e.g. curses) in the process output can make matching difficult.
  • The version of regex currently used mostly follows Google's RE2.
  • The matching of the process output is not affected by the quiet script option.

Global (Cogment >= 2.15)​

The global node contains values that have a global scope (i.e. they affect all scripts defined in the file). Some of these values can be overridden by the individual scripts.

Environment Variables​

You can set global environment variables using the environment node under the global node. These variables will be part of the environment of all scripts, unless overridden locally in the script's own environment node.

E.g:

global:
    environment:
        COGMENT_LOG_LEVEL: info
        COGMENT_DIRECTORY_ENDPOINT: grpc://server:9010
Working folder​

By default, the current working folder is set to the folder containing the launch definition file (not the folder where the Launch command is executed). This can be changed with the folder node under the global node. It will then become the default folder for all scripts, unless locally changed by the script's own folder node. Each level can also build on the previous one if the provided folder is a relative folder.

E.g.:

# Definition file is in "/home/user"
global:
    folder: ./app # Default folder for all scripts is set to "/home/user/app"
scripts:
    local:
        folder: ./inside # Working folder is then "/home/user/app/inside"
    somewhere:
        folder: /home/away # Working folder is then "/home/away"

Variable substitution​

You can substitute launch variables using {{.VAR}} in strings anywhere in the commands, environment or ready_output nodes of the yaml definition file. All environment variables when launch is started are defined as launch variables. New environment variables set in the definition file also define launch variables. And launch defines some special variables internally also.

In the concerned strings, the double open curly brackets ("{{") delimit the start of a variable to be substituted, therefore to include a literal double open brackets in a string, you have to surround them with backticks inside substitution brackets: {{`{{`}}.

Undefined values will be replaced with <no value>.

E.g.:

$ export OWNER=Elvis
$ cogment launch ./launch.yaml
scripts:
    say_hi:
        environment:
            Type: "How"
            Question: "{{.Type}} are you?"
        commands:
            - ["echo", "Hello, {{.OWNER}}. {{.Question}}"] # echo "Hello, Elvis. How are you?"
            - ["echo", "{{`{{`}} brackets }}"] # echo "{{ brackets }}"
            - ["echo", "no val: {{.MADE_UP_VARIABLE}}"] # echo "no val: <no value>"
        ready_output: "Hello, {{.OWNER}}.*"

Special Variables​

The launch program will also define variables that can be used in substitutions. These internally defined variables start with a double underscore (__).

Environment variables with conflicting names will be replaced by these internal launch variables. Note that this only affects variable substitution, not the environment of the command execution. And internal launch variables are not available as environment variables.

Arguments (Cogment >= 2.15)​

The arguments from the command line of launch define variables and can thus be substituted. E.g. cogment launch ./launch.yaml arg1 arg2 arg3 will define __1, __2 and __3 respectively corresponding to arg1, arg2 and arg3. They can then be substituted with {{.__1}}, {{.__2}}, {{.__3}}.

__1 to __9 are always defined, but will be empty if no corresponding argument was given on the command line. Arguments 10 (__10) and above will only be defined if they were provided on the command line.

E.g.:

$ cogment launch -q ./launch.yaml 42 foo
scripts:
    args_out:
        commands:
            - ["echo", "args:", "{{.__1}}", "{{.__2}}", "{{.__3}}"] # echo "args:" "42" "foo" ""
            - ["echo", "empty: >{{.__3}}< >{{.__6}}<"] # echo "empty: >< ><"
            - ["echo", "no args: {{.__10}} {{.__42}}"] # echo "no args: <no value> <no value>"
        ready_output: '.* {{.__2}} $'
All Arguments (Cogment >= 2.17)​

The number of arguments on the command line of launch is defined in __NB_ARGS. And all the launch arguments can be added to a script command with __ALL_ARGS.

The __ALL_ARGS variable is special in terms of substitution; if the command argument string is exactly {{.__ALL_ARGS}}, then all launch arguments will be added to this command. Otherwise if it is only part of a string, it will substitute a string containing all launch arguments.

In other words, a command argument "{{.ALL_ARGS}}" will expand to multiple arguments for the command, whereas something like "--{{.ALL_ARGS}}" will stay as one argument (a string that contains all launch arguments preceded by "--").

E.g.:

$ cogment launch ./launch.yaml 42 foo extra
scripts:
    args_out:
        commands:
            - ["echo", "nb of args: {{.__NB_ARGS}}]"] # echo "nb of args: 3"
            - ["echo", "all:", "{{.__ALL_ARGS}}", "and more"] # echo "all:" "42" "foo" "extra" "and more"
            - ["echo", "all:", "args: {{.__ALL_ARGS}}"] # echo "all:" "args: 42 foo extra"
        ready_output: 'args:.*{{.__2}}'

File Example​

global:
    environment:
        RUN_NAME: "COGRUN-{{.HOSTNAME}}-{{.__1}}"
        COGMENT_LOG_LEVEL: info
        DIR_PORT: 9010
        COGMENT_DIRECTORY_ENDPOINT: "grpc://server:{{.DIR_PORT}}"
    folder: /app

scripts:
    prep:
        commands:
            - ["./importdata.sh"]
            - ["./cudasetup.sh"]
            - ["sleep", "infinity"] # To prevent process from ending
        ready_output: '^Done'
    directory:
        quiet: true
        folder: ./cogment
        environment:
            COGMENT_DIRECTORY_PORT: "{{.DIR_PORT}}"
        depends_on:
            - prep
        commands:
            - ["cogment", "services", "directory"]
        ready_output: 'Listening \[port:[0-9]*\]' # To match "Listening [port:9010]"
    orchestrator:
        folder: ./cogment
        environment:
            COGMENT_ORCHESTRATOR_ACTOR_PORT: 0
            COGMENT_LIFECYCLE_PORT: 0
        depends_on:
            - directory
        commands:
            - ["cogment", "services", "orchestrator"]
    runner:
        folder: ./pycode
        depends_on:
            - directory
            - orchestrator
        commands:
            - [
                  "python3",
                  "runner.py",
                  "-u {{.USER}}",
                  "-n {{.RUN_NAME}}",
                  "{{.__2}}",
              ]
    report:
        commands:
            - ["reporting_service"]
Last updated on