Set up the virtual environment
Once you have set up drunc with or without DUNE-DAQ, in the working directory root run
Running drunc
This can currently be done either with drunc-unified-shell, which is recommended for first time/casual users, or with a process-manager-shell and controller-shell, which is recommended for expert developers.
All of the drunc shells support tab completion.
With drunc-unified-shell
Spawning the shell and process_manager
In your terminal window run drunc-unified-shell as
<process_manager_configuration> can be
-
ssh-standalone.json- ssh based standalone implementation without a kafka feed. -
ssh-kafka.json- ssh based implementation with kafka for message broadcasting. -
ssh-CERN-kafka.json- ssh based implementation with kafka service running at ENH1. -
k8s.json- kubernetes implementation (not recommended unless you are an expert user). Note that if you are outside of cern you are limited tossh-standaloneandssh-kafka.
Interacting with process_manager
At this point the process_manager has been spawned, and you can interface it directly through the current shell. You can now spawn the DAQ processes as
ps as
bash
drunc-unified-shell > ps
Processes running
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓
┃ session ┃ friendly name ┃ user ┃ host ┃ uuid ┃ alive ┃ exit-code ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩
│ test-session │ root-controller │ pplesnia │ localhost │ cb986b26-221c-4819-bc9b-20825b11bee9 │ True │ 0 │
│ test-session │ local-connection-server │ pplesnia │ localhost │ 0994f364-8d1f-4ccc-98f1-1872f8722b0f │ True │ 0 │
│ test-session │ ru-controller │ pplesnia │ localhost │ 2715c09e-d568-4e86-80a2-bb7827fd46fc │ True │ 0 │
│ test-session │ ru-01 │ pplesnia │ localhost │ d93f7922-377e-463f-8b26-523b73fb7d6b │ True │ 0 │
│ test-session │ df-controller │ pplesnia │ localhost │ d91c71f7-3c41-4a7e-aab0-7fb98d99c071 │ True │ 0 │
| ... | ... | ... | ... | ... | ... | ... |
└──────────────┴───────────────────────────┴──────────┴───────────┴──────────────────────────────────────┴───────┴───────────┘
We have started a couple of processes - the standard DAQ applications organized into segments each with a controller, and a root-controller that will control all the segments. The logs and work directory is the ${PWD} where you executed the drunc-unified-shell. From here you can use the process_manger commands
-
ps- list all the processes -
kill- kill specific processes -
flush- remove dead processes -
restart- restart processes -
logs- show the logs of specific processes -
terminate- kill all the processes
There are many more functionalities to the shell, head over to the process manager documentation to see how to interact with it.
Interacting with root-controller
Next, let's send commands to the root-controller. These commands will be propagated by it to other applications using gRPC. To see which segments a controller controls, you can use ls:
bash
drunc-unified-shell > ls
['ru-controller', 'df-controller', 'trg-controller', 'hsi-controller']
root-controller are
-
describe- shows available endpoint commands (not this currently describes what the controller endpoint can do internally not through the shell) -
ls- lists all running segments -
status- lists the FSM state, substate, error status, and included parameters of the segments -
connect- remaps the root controller -
take_control- updates the user in charge (actor) -
surrender_control- releases the current actor -
who_am_i- prints your username -
who_is_in_charge- prints who is in charge of the root controller -
include- includes self in the current session -
exclude- excludes self from the current session -
fsm <fsm-command> <fsm-command-args>- executes finite state machine (FSM) commands. These are explanied more here.
FSM commands can be executed directly from the shell with the following commands:
-
fsm conf- configure the applications by ingesting the parameters from the configuration file to the applications -
fsm start- start a run, allocating a run number. Initializes queues and connections -
fsm enable_triggers- start generating TPs, TDs are not propagated to the DFO -
fsm disable_triggers- stop collecting generated TPs to file -
fsm drain_dataflow- stop propagating TDs to the TRBs. -
fsm stop_trigger_sources- stop generating TPs -
fsm stop- stop app communication -
fsm scrap- remove all the configuration parameters from the applications Note that some of these commands require arguments, see here for details.
Operating a DAQ with drunc
Let's take the DAQ for a spin:
bash
drunc-unified-shell > fsm conf
drunc-unified-shell > fsm start 12345 # start the run 12345
drunc-unified-shell > fsm enable-triggers
[...we wait for a bit of time, to get a file...]
drunc-unified-shell > fsm disable-triggers
drunc-unified-shell > fsm drain-dataflow
drunc-unified-shell > fsm stop-trigger-sources
drunc-unified-shell > fsm stop
drunc-unified-shell > fsm scrap
Several things to note:
-
There is a profusion of logging that happens. This is coming from the asynchronous logging from the controller. If someone tries to connect at the same time or to execute a command you will see it appearing on your shell too (you can try it yourself in a different shell). Unfortunately CLIs do not lend itself very well with this, and one needs a better UI for the logs to be rendered better and to not distract.
-
The
describecommand describes the endpoint NOT the shell. This means that there are some commands that are not directly available in the shell (for exampleget_children_status). However if you dostatusis the shell, you will get the children statuses because the shell callsget_children_statusunder the hood.
You can now head to the controller documentation for more information.
Shutting down
orbash
drunc-unified-shell > terminate
drunc-unified-shell > quit
```# Set up the virtual environment
Once you have set up `drunc` [with](https://DUNE-DAQ.github.io/drunc/developer-documentation/Setup/Setup-drunc-with-DUNE-DAQ) or [without](https://DUNE-DAQ.github.io/drunc/Setup-drunc-without-DUNE-DAQ) DUNE-DAQ, in the working directory root run
to make sure all the environment variables are correctly set. This only necessary if you are returning to a work directory.
# Running `drunc`
This can currently be done either with `drunc-unified-shell`, which is recommended for first time/casual users, or with a `process-manager-shell` and `controller-shell`, which is recommended for expert developers.
All of the `drunc` shells support tab completion.
## With `drunc-unified-shell`
### Spawning the shell and `process_manager`
In your terminal window run `drunc-unified-shell` as
For which `<process_manager_configuration>` can be
- `ssh-standalone.json` - ssh based standalone implementation without a kafka feed.
- `ssh-kafka.json` - ssh based implementation with kafka for message broadcasting.
- `ssh-CERN-kafka.json` - ssh based implementation with kafka service running at ENH1.
- `k8s.json` - kubernetes implementation (not recommended unless you are an expert user).
Note that if you are outside of cern you are limited to `ssh-standalone` and `ssh-kafka`.
### Interacting with `process_manager`
At this point the `process_manager` has been spawned, and you can interface it directly through the current shell. You can now spawn the DAQ processes as
An example of the expected output is provided [here](https://DUNE-DAQ.github.io/drunc/Sample-outputs#for-running-process_manager-boot). You can then check the running processes with `ps` as
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓ ┃ session ┃ friendly name ┃ user ┃ host ┃ uuid ┃ alive ┃ exit-code ┃ ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩ │ test-session │ root-controller │ pplesnia │ localhost │ cb986b26-221c-4819-bc9b-20825b11bee9 │ True │ 0 │ │ test-session │ local-connection-server │ pplesnia │ localhost │ 0994f364-8d1f-4ccc-98f1-1872f8722b0f │ True │ 0 │ │ test-session │ ru-controller │ pplesnia │ localhost │ 2715c09e-d568-4e86-80a2-bb7827fd46fc │ True │ 0 │ │ test-session │ ru-01 │ pplesnia │ localhost │ d93f7922-377e-463f-8b26-523b73fb7d6b │ True │ 0 │ │ test-session │ df-controller │ pplesnia │ localhost │ d91c71f7-3c41-4a7e-aab0-7fb98d99c071 │ True │ 0 │ | ... | ... | ... | ... | ... | ... | ... | └──────────────┴───────────────────────────┴──────────┴───────────┴──────────────────────────────────────┴───────┴───────────┘
The full expected output is shown [here](https://DUNE-DAQ.github.io/drunc/Sample-outputs#checking-the-processes-with-ps).
We have started a couple of processes - the standard DAQ applications organized into segments each with a controller, and a `root-controller` that will control all the segments. The logs and work directory is the `${PWD}` where you executed the `drunc-unified-shell`. From here you can use the `process_manger` commands
- `ps` - list all the processes
- `kill` - kill specific processes
- `flush` - remove dead processes
- `restart` - restart processes
- `logs` - show the logs of specific processes
- `terminate` - kill all the processes
There are many more functionalities to the shell, head over to the [process manager documentation](https://DUNE-DAQ.github.io/drunc/developer-documentation/Operation/Process-manager) to see how to interact with it.
### Interacting with `root-controller`
Next, let's send commands to the `root-controller`. These commands will be propagated by it to other applications using `gRPC`. To see which segments a controller controls, you can use `ls`:
The set of commands that you can send to the `root-controller` are
- `describe` - shows available _endpoint_ commands (not this currently describes what the controller endpoint can do internally not through the shell)
- `ls` - lists all running segments
- `status` - lists the FSM state, substate, error status, and included parameters of the segments
- `connect` - remaps the root controller
- `take_control` - updates the user in charge (actor)
- `surrender_control` - releases the current actor
- `who_am_i` - prints your username
- `who_is_in_charge` - prints who is in charge of the root controller
- `include` - includes self in the current session
- `exclude` - excludes self from the current session
- `fsm <fsm-command> <fsm-command-args>` - executes finite state machine (FSM) commands. These are explanied more [here](https://DUNE-DAQ.github.io/drunc/developer-documentation/Operation/FSM).
[FSM](https://DUNE-DAQ.github.io/drunc/developer-documentation/Operation/FSM) commands can be executed directly from the shell with the following commands:
- `fsm conf` - configure the applications by ingesting the parameters from the configuration file to the applications
- `fsm start` - start a run, allocating a run number. Initializes queues and connections
- `fsm enable_triggers` - start generating TPs, TDs are not propagated to the DFO
- `fsm disable_triggers` - stop collecting generated TPs to file
- `fsm drain_dataflow` - stop propagating TDs to the TRBs.
- `fsm stop_trigger_sources` - stop generating TPs
- `fsm stop` - stop app communication
- `fsm scrap` - remove all the configuration parameters from the applications
Note that some of these commands require arguments, see [here](https://DUNE-DAQ.github.io/drunc/developer-documentation/Operation/FSM) for details.
## Operating a DAQ with `drunc`
Let's take the DAQ for a spin:
Several things to note:
- There is a profusion of logging that happens. This is coming from the asynchronous logging from the controller. If someone tries to connect at the same time or to execute a command you will see it appearing on your shell too (you can try it yourself in a different shell). Unfortunately CLIs do not lend itself very well with this, and one needs a better UI for the logs to be rendered better and to not distract.
- The `describe` command describes the _endpoint_ **NOT** the shell. This means that there are some commands that are not directly available in the shell (for example `get_children_status`). However if you do `status` is the shell, you will get the children statuses because the shell calls `get_children_status` under the hood.
You can now head to the [controller documentation](https://DUNE-DAQ.github.io/drunc/developer-documentation/Operation/Controller) for more information.
# Shutting down