Metadata-Version: 2.4
Name: vigie
Version: 0.0.1
Summary: VM Migration Scheduler for OpenStack
Home-page: https://www.openstack.org/
Author: OpenStack
Author-email: openstack-discuss@lists.openstack.org
Classifier: Environment :: OpenStack
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Dist: pbr>=5.7.0
Requires-Dist: Flask>=2.0.0
Requires-Dist: oslo.config>=8.7.0
Requires-Dist: oslo.context>=4.1.0
Requires-Dist: oslo.db>=12.1.0
Requires-Dist: oslo.log>=4.1.0
Requires-Dist: oslo.middleware>=5.0.0
Requires-Dist: oslo.messaging>=12.5.2
Requires-Dist: oslo.policy>=3.5
Requires-Dist: oslo.service>=3.0.0
Requires-Dist: oslo.utils>=4.8.0
Requires-Dist: keystoneauth1>=4.2
Requires-Dist: keystonemiddleware>=9.1.0
Requires-Dist: openstacksdk>=0.101.0
Requires-Dist: alembic>=1.7.0
Requires-Dist: sqlalchemy>=1.4.0
Requires-Dist: tooz>=2.7.1
Requires-Dist: vigietools>=0.0.10
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: home-page
Dynamic: requires-dist
Dynamic: summary

# Table of Contents

- [What is Vigie](#what-is-vigie)
  * [General description](#general-description)
  * [Features](#features)
- [Installation](#installation)
  * [Package](#package)
  * [Configuration](#configuration)
- [Usage](#usage)
  * [Startup](#startup)
  * [Vigie client](#vigie-client)
  * [API Endpoints](#api-endpoints)
- [Internal code structure](#internal-code-structure)
  * [Inheritance from Cinder](#inheritance-from-cinder)
  * [Writing a new API function](#writing-a-new-api-function)
  * [Writing a new worker task](#writing-a-new-worker-task)
  * [Using vigietools](#using-vigietools)

# What is Vigie

## General description

Vigie is an OpenStack operator API aimed at easing the administration of
an existing deployment. It is typically used by operators to do daily
tasks such as purging all resources of a project, or evacuating a
compute or a network node for maintenance, and more.

WARNING: this project is in its early stages.

<img src="doc/source/_static/vigie.png" alt="Vigie" width="250" />

## Features

### Currently implemented

* Validating a compute node by spawning a VM on it and connecting to it
  over SSH

### Planned features

* Public image management (downloads from distro repos)
* Project pre-delete
* Project purge of resources
* Emptying / evacuation of a compute node (live or cold migration,
  depending on whether the VMs support it)
* Emptying / evacuation of a network node (gracefully, with conntrack)

# Installation

## Package

Use the Debian package or invent a deployment method yourself.

## Configuration

The service requires a configuration file typically located at
``/etc/vigie/vigie.conf``.

Example configuration:

```
    [DEFAULT]
    transport_url=rabbit://vigie:PASSWORD@cl1-controller-3.example.com:5671
    [database]
    connection=sqlite:///var/lib/vigie/vigie.db

    [keystone_authtoken]
    www_authenticate_uri = http://keystone:5000
    auth_url = http://keystone:5000
    auth_type = password
    project_domain_name = Default
    user_domain_name = Default
    project_name = service
    username = vigie
    password = vigie_password

    [validate_compute]
    glance_image=debian-13-generic-amd64-daily.qcow2
    flavor=cpu1-ram2-disk5
    network=ext-net1
    ssh_username=debian
    keep_enabled=True
    auto_clean_up=True
    validation_timeout=600
    ssh_use_jump_host=False
    ssh_jump_host_hostname=jump.example.com
    ssh_jump_host_key_file=/etc/vigie/.ssh/id_rsa

    [worker]
    coordination_url=zookeeper://192.168.0.2:2181,192.168.0.66:2181,192.168.0.130:2181/

    [tasks]
    task_auto_clean=True
    task_auto_clean_after=1440
```

At startup, the worker daemon joins the coordination backend configured
through ``[worker]/coordination_url`` and checks that it can actually
acquire and release a tooz lock. When the option is not set, or when
the backend does not provide working locks, the worker daemon refuses
to start: cluster-wide locks are required for its features.

Before deciding whether a validation or a clean-up may run, a worker
takes a cluster-wide tooz lock, reads the tasks table and switches the
task to its new state (RUNNING or WAITING) while still holding it, and
only then releases the lock: the decision is atomic across the
workers, and two of them can never start a validation at the same
time by both reading that no validation is running. If the lock
cannot be acquired, nothing is written to the task. When the
coordination backend cannot be reached at all, the decision is not
taken either: the task is left untouched, so that an infrastructure
problem never alters its state. The lock is released as soon as the
decision is taken: the validation itself runs without it.

When ``[tasks]/task_auto_clean`` is enabled (the default), the
worker's periodic tasks purge every task which has been in the DONE
state for longer than ``[tasks]/task_auto_clean_after`` minutes
(default: 1440, that is one day). The purge runs under a cluster-wide
tooz lock, so that only one worker purges at a time. Tasks which are
not DONE are never purged, so that a failed validation can still be
inspected. They are instead deleted by hand, with the tasks-clean
endpoint.

When ``[validate_compute]/auto_clean_up`` is enabled (the default), the
worker automatically schedules a clean-up task once all the compute
validations are DONE: a validation which is still scheduled, waiting or
running holds it back, and so does a validation in ERROR, so that its
leftovers can be inspected first. The automatic clean-up is dispatched
to the worker which just finished the last validation, which is by
construction the only one free of validations at that point.

# Usage

## Startup

Start the API service using uwsgi.

Start the worker daemon:

```
vigie-worker
```

The Debian package of course comes with systemd .service unit files,
so it is possible to do:

```
systemctl restart vigie-api
systemctl restart vigie-worker
```

At least one worker is needed, though it is best, for redundancy, to
provision three of them. Same for the API.

## Vigie client

Then you may use the vigieclient, which is able to validate one or
many compute nodes with a single request:

```
openstack vigie validate compute node1.example.com node2.example.com
```

## API Endpoints

* ``POST /v2/validate-compute`` - Validate one or more compute nodes
  by spawning a VM on each of them. The request body is a JSON object
  with a non-empty ``hostnames`` list, for example:
  ``{"hostnames": ["node1.example.com", "node2.example.com"]}``. The
  optional boolean ``keep_enabled`` member overrides, for this call,
  the ``[validate_compute]/keep_enabled`` configured default. Note
  that even with ``keep_enabled=true``, the host will be disabled
  again if the validation fails. Every hostname is checked before
  anything is scheduled: should any of them be invalid, unknown to
  Nova, or have its nova-compute service down, the whole request is
  rejected with a 400, 404 or 409 status, and no task is created at
  all. Otherwise, one task per hostname is created, all dispatched to
  a single worker, and the 202 response carries the created task IDs.
* ``POST /v2/validate-cleanup`` - Clean up all the resources left behind
  by previous validations (the hosts-to-validate aggregate, the
  validation trait, and the whole vgt project with everything in it).
  The clean-up is preferably dispatched to an alive worker without any
  remaining validation task, and falls back to any alive worker
  otherwise.
* ``POST /v2/tasks-clean`` - Delete all the tasks which are DONE.
* ``POST /v2/evacuate-compute`` - Evacuate a compute node.
* ``POST /v2/purge-project`` - Purge a project.

# Internal code structure

## Inheritance from Cinder

Having read a lot of OpenStack code, I have found that the Manager
class from Cinder is really nice, so I used it as an example for
writing vigie/manager.py. The worker in vigie/worker/manager.py is a
subclass of that one.

## Writing a new API function

The Vigie API uses Flask, as it is simple, understandable, and the
most commonly used framework.

To write a new API route, simply cheat on one of the existing ones.
For example:

```
@app.route('/v2/validate-compute', methods=['POST'])
@policy_checker('vigie:validate_compute')
def validate_compute():
```

Simply using the @policy_checker is enough provided that such a
policy is written in vigie/policy/rules.py. The @app.route is
handled by Flask.

## Writing a new worker task

Each worker task lives in its own module of the vigie/worker/tasks/
package, as a plain mixin class holding the methods of one category
of tasks. The only registration needed is inheritance: add the new
class to the inheritance list of the WorkerManager in
vigie/worker/manager.py, and its public methods become reachable
through RPC (an incoming RPC message is dispatched to the public
method of the manager matching the method name in the message;
private methods, with a leading underscore, are never reachable).

The signature of an RPC method is: the context (``ctxt``) as the
first positional argument, followed by the keyword arguments sent by
the caller. For example, in vigie/worker/tasks/evacuate_compute.py:

```
def evacuate_compute(self, ctxt, source_hostname, target_hostname=None):
```

the caller casts it with:

```
client.cast({}, 'evacuate_compute', source_hostname=...,
            target_hostname=...)
```

so the method name and the keyword arguments must match on both
sides: that is the whole RPC contract.

When the task must never run concurrently with another task of the
same group on the whole cluster, decorate it with
``@vigie_worker_mutually_exclusive(worker_group_name='my-group')``
and add a matching ``WorkerGroup`` entry in
vigie/worker/tasks/base.py. The decorator then changes the RPC
signature: the caller only sends a ``task_id`` plus the task
arguments, and the decorator fetches the task, opens the database
session, and waits for a free execution slot before calling the
method with the task and the session injected:

```
@vigie_worker_mutually_exclusive(worker_group_name='validate-compute')
def validate_compute(self, ctxt, task, session, compute_hostname,
                     project_id, project_name, keep_enabled=None,
                     req_id=None):
```

Over RPC, this is called with ``ctxt``, ``task_id`` and the remaining
keyword arguments; ``task`` and ``session`` are provided by the
decorator.

A mixin may also hold periodic tasks, simply decorated with the
standard ``@periodic_task.periodic_task`` decorator from
oslo_service: they are collected from the mixins by
``vigie.manager.register_mixin_periodic_tasks()``, which is already
called at the bottom of vigie/worker/manager.py, so they need no
extra wiring either. As always, cheat on an existing task (for
example vigie/worker/tasks/validate_compute.py) and its unit tests
when writing a new one.

## Using vigietools

Vigie is mostly a scheduler for vigietools. For example, the
validate-compute task calls a vigietools function to perform the
task. The advantage is that vigietools itself also contains a command
line utility (i.e., /usr/bin/vgt) that can be used standalone,
without the Vigie task manager.

Vigie, with its task manager, provides orchestration and
asynchronous tasks, which would not be possible with vigietools
alone.

