Welcome to Estuary Updater’s documentation!¶

Getting Started¶

Overview¶

Estuary Updater is a micro-service that updates the Neo4j graph database used by Estuary in real-time by reading and processing messages from the UMB.

Run the Unit Tests¶

Since the unit tests require a running Neo4j instance, the tests are run in Docker containers using Docker Compose. The commands required to run the unit tests are abstracted in scripts/run-tests.sh. This script will create the Docker image required to run the tests based on docker/Dockerfile-tests, create a container with Neo4j, create another container to run the tests based on the built Docker image, run the tests, and then delete the two created containers.

To install Docker and Docker Compose on Fedora, run:

$ sudo dnf install docker docker-compose

To start Docker, run:

$ sudo systemctl start docker

To run the tests, run:

$ sudo scripts/run-tests.sh

To run just a single test, you can run:

sudo scripts/run-tests.sh pytest-3 -vvv tests/test_file::test_name

Code Styling¶

The codebase conforms to the style enforced by flake8 with the following exceptions:

The maximum line length allowed is 100 characters instead of 80 characters

In addition to flake8, docstrings are also enforced by the plugin flake8-docstrings with the following exemptions:

D100: Missing docstring in public module
D104: Missing docstring in public package

The format of the docstrings should be in the Sphynx style such as:

Get a node from Neo4j.

:param str resource: a resource name that maps to a neomodel class
:param str uid: the value of the UniqueIdProperty to query with
:return: an object representing the Neo4j node
:rtype: neomodel.StructuredNode
:raises ValidationError: if an invalid resource was requested

Creating a New Handler¶

Create a new file in estuary_updater/handlers such as estuary_updater/handlers/distgit.py.

At the top of the new file, add the following license header and imports:

# SPDX-License-Identifier: GPL-3.0+

from __future__ import unicode_literals, absolute_import

from estuary_updater.handlers.base import BaseHandler

Then you can proceed to create your handler in the same file as such:

class DistGitHandler(BaseHandler):
"""A handler for dist-git related messages."""

@staticmethod
def can_handle(msg):
    """
    Determine if this handler can handle this message.

    :param dict msg: a message to be analyzed
    :return: a bool based on if the handler can handle this kind of message
    :rtype: bool
    """
    # Code goes here to determine if the message can be handled by this handler

def handle(self, msg):
    """
    Handle a message and update Neo4j if necessary.

    :param dict msg: a message to be processed
    """
    # Code goes here to handle/process the message

Then register your handler by adding the class to estuary_updater.handlers.all_handlers such as:

from estuary_updater.handlers.distgit import DistGitHandler

all_handlers = [DistGitHandler, OtherHandlerHere]

Lastly, add any additional topics to the fedmsg.d/config.py file by editing the estuary_updater.topics value.

Writing a New Unit Test For Your Handler¶

Create a new file to store the JSON message you want to test your handler with. This should be stored in tests/messages such as tests/messages/distgit/new_commit.json.
Create a new file in tests/handlers/ such as tests/handlers/distgit.py.

At the top of the new file, add the following license header and imports:

# SPDX-License-Identifier: GPL-3.0+

from __future__ import unicode_literals, absolute_import

import json
from os import path

Then you can proceed to create your unit test in the same file as such:

from estuary.models.distgit import DistGitCommit

from tests import message_dir
from estuary_updater.handlers.distgit import DistGitHandler
from estuary_updater import config


def test_distgit_new_commit():
    """Test the dist-git handler when it recieves a new commit message."""
    # Load the message to pass to the handler
    with open(path.join(message_dir, 'distgit', 'new_commit.json'), 'r') as f:
        msg = json.load(f)
    # Make sure the handler can handle the message
    assert DistGitHandler.can_handle(msg) is True
    # Instantiate the handler
    handler = DistGitHandler(config)
    # Run the handler
    handler.handle(msg)
    # Verify everything looks good in Neo4j
    commit = DistGitCommit.nodes.get_or_none(hash_='some_hash_from_the_message')
    assert commit is not None
    # Do additional checks here

Code Documentation¶

To document new files, please check here.

Handlers¶

Base¶

class estuary_updater.handlers.base.BaseHandler(config)[source]¶

An abstract base class for handlers to enforce the API.

static can_handle(msg)[source]¶

Determine if this handler can handle this message.

Parameters:	msg (dict) – a message to be analyzed
Returns:	a bool based on if the handler can handle this kind of message
Return type:	bool

get_or_create_build(identifier, original_nvr=None, force_container_label=False)[source]¶

Get a Koji build from Neo4j, or create it if it does not exist in Neo4j.

Parameters:	identifier (str/int) – an NVR (str) or build ID (int), or a dict of info from Koji API original_nvr (str) – original_nvr property for the ContainerKojiBuild force_container_label (bool) – when true, this skips the check to see if the build is a container and just creates the build with the ContainerKojiBuild label
Return type:	KojiBuild
Returns:	the Koji Build retrieved or created from Neo4j

handle(msg)[source]¶

Handle a message and update Neo4j if necessary.

Parameters:	msg (dict) – a message to be processed

is_container_build(build_info)[source]¶

Check whether a Koji build is a container build.

Parameters:	build_info (KojiBuild) – build info from the Koji API
Returns:	boolean value indicating whether the build is a container build
Return type:	bool

is_module_build(build_info)[source]¶

Check whether a Koji build is a module build.

Parameters:	build_info (KojiBuild) – build info from Koji API
Returns:	boolean value indicating whether the build is a module build
Return type:	bool

koji_session¶

Get a cached Koji session but initialize the connection first if needed.

Returns:	a Koji session object
Return type:	koji.ClientSession

DistGit¶

class estuary_updater.handlers.distgit.DistGitHandler(config)[source]¶

A handler for dist-git related messages.

static can_handle(msg)[source]¶

Determine if this is a dist-git message.

Parameters:	msg (dict) – a message to be analyzed
Returns:	a bool based on if the handler can handle this kind of message
Return type:	bool

commit_handler(msg)[source]¶

Handle a dist-git commit message and update Neo4j if necessary.

Parameters:	msg (dict) – a message to be processed

handle(msg)[source]¶

Handle a dist-git message by sending it to the right handler method and update Neo4j.

Parameters:	msg (dict) – a message to be processed

static parse_bugzilla_bugs(commit_message)[source]¶

Parse the Bugzilla bugs mentioned in a a dist-git commit message.

Parameters:	commit_message (str) – the dist-git commit message
Return type:	dict
Returns:	a dictionary with the keys resolves, related, reverted with values as lists of Bugzilla IDs

Errata¶

class estuary_updater.handlers.errata.ErrataHandler(config)[source]¶

A handler for dist-git related messages.

advisory_handler(msg)[source]¶

Handle an Errata tool advisory changes and update Neo4j if necessary.

Parameters:	msg (dict) – a message to be processed

builds_added_handler(msg)[source]¶

Handle an Errata tool builds added message and update Neo4j if necessary.

Parameters:	msg (dict) – a message to be processed

builds_removed_handler(msg)[source]¶

Handle an Errata tool builds removed message and update Neo4j if necessary.

Parameters:	msg (dict) – a message to be processed

static can_handle(msg)[source]¶

Determine if this handler can handle this message.

Parameters:	msg (dict) – a message to be analyzed
Returns:	a bool based on if the handler can handle this kind of message
Return type:	bool

handle(msg)[source]¶

Handle an Errata tool message and update Neo4j if necessary.

Parameters:	msg (dict) – a message to be processed

Freshmaker¶

class estuary_updater.handlers.freshmaker.FreshmakerHandler(config)[source]¶

A handler for Freshmaker-related messages.

build_state_handler(msg)[source]¶

Handle a Freshmaker build state changed message and update Neo4j if necessary.

Parameters:	msg (dict) – a message to be processed

static can_handle(msg)[source]¶

Determine if the message is a Freshmaker-related message and can be handled by this handler.

Parameters:	msg (dict) – a message to be analyzed
Returns:	a bool based on if the handler can handle this kind of message
Return type:	bool

create_or_update_build(build, event_id)[source]¶

Use the Koji Task Result to create or update a ContainerKojiBuild.

Parameters:	build (dict) – the build represented in Freshmaker being created or updated event_id (int) – the id of the Freshmaker event
Returns:	the created/updated ContainerKojiBuild or None if it cannot be created
Return type:	ContainerKojiBuild or None

create_or_update_freshmaker_build(build, event_id)[source]¶

Create or update a FreshmakerBuild.

Parameters:	build (dict) – the build represented in Freshmaker being created or updated event_id (int) – the id of the Freshmaker event
Returns:	the created/updated FreshmakerBuild or None if it cannot be created
Return type:	FreshmakerBuild or None

event_state_handler(msg)[source]¶

Handle a Freshmaker event state changed message and update Neo4j if necessary.

Parameters:	msg (dict) – a message to be processed

handle(msg)[source]¶

Handle a Freshmaker message and update Neo4j if necessary.

Parameters:	msg (dict) – a message to be processed

Koji¶

class estuary_updater.handlers.koji.KojiHandler(config)[source]¶

A handler for Koji related messages.

build_handler(msg)[source]¶

Handle a build state message and update Neo4j if necessary.

Parameters:	msg (dict) – a message to be processed

static can_handle(msg)[source]¶

Determine if this handler can handle this message.

Parameters:	msg (dict) – a message to be analyzed
Returns:	a bool based on if the handler can handle this kind of message
Return type:	bool

handle(msg)[source]¶

Handle a message and update Neo4j if necessary.

Parameters:	msg (dict) – a message to be processed

Welcome to Estuary Updater’s documentation!¶

Getting Started¶

Overview¶

Run the Unit Tests¶

Code Styling¶

Creating a New Handler¶

Writing a New Unit Test For Your Handler¶

Code Documentation¶

Handlers¶

Base¶

DistGit¶

Errata¶

Freshmaker¶

Koji¶

Indices and tables¶