Welcome to Estuary Updater’s documentation!¶
Getting Started¶
Overview¶
Estuary Updater is a micro-service that updates the Neo4j graph database used by Estuary in real-time by reading and processing messages from the UMB.
Run the Unit Tests¶
Since the unit tests require a running Neo4j instance, the tests are run in Docker containers using
Docker Compose. The commands required to run the unit tests are abstracted in
scripts/run-tests.sh
. This script will create the Docker image required to run the tests based
on docker/Dockerfile-tests
, create a container with Neo4j, create another container to run
the tests based on the built Docker image, run the tests, and then delete the two created
containers.
To install Docker and Docker Compose on Fedora, run:
$ sudo dnf install docker docker-compose
To start Docker, run:
$ sudo systemctl start docker
To run the tests, run:
$ sudo scripts/run-tests.sh
To run just a single test, you can run:
sudo scripts/run-tests.sh pytest-3 -vvv tests/test_file::test_name
Code Styling¶
The codebase conforms to the style enforced by flake8
with the following exceptions:
- The maximum line length allowed is 100 characters instead of 80 characters
In addition to flake8
, docstrings are also enforced by the plugin flake8-docstrings
with
the following exemptions:
- D100: Missing docstring in public module
- D104: Missing docstring in public package
The format of the docstrings should be in the Sphynx style such as:
Get a node from Neo4j.
:param str resource: a resource name that maps to a neomodel class
:param str uid: the value of the UniqueIdProperty to query with
:return: an object representing the Neo4j node
:rtype: neomodel.StructuredNode
:raises ValidationError: if an invalid resource was requested
Creating a New Handler¶
Create a new file in
estuary_updater/handlers
such asestuary_updater/handlers/distgit.py
.At the top of the new file, add the following license header and imports:
# SPDX-License-Identifier: GPL-3.0+ from __future__ import unicode_literals, absolute_import from estuary_updater.handlers.base import BaseHandler
Then you can proceed to create your handler in the same file as such:
class DistGitHandler(BaseHandler): """A handler for dist-git related messages.""" @staticmethod def can_handle(msg): """ Determine if this handler can handle this message. :param dict msg: a message to be analyzed :return: a bool based on if the handler can handle this kind of message :rtype: bool """ # Code goes here to determine if the message can be handled by this handler def handle(self, msg): """ Handle a message and update Neo4j if necessary. :param dict msg: a message to be processed """ # Code goes here to handle/process the message
Then register your handler by adding the class to
estuary_updater.handlers.all_handlers
such as:from estuary_updater.handlers.distgit import DistGitHandler all_handlers = [DistGitHandler, OtherHandlerHere]
Lastly, add any additional topics to the
fedmsg.d/config.py
file by editing theestuary_updater.topics
value.
Writing a New Unit Test For Your Handler¶
Create a new file to store the JSON message you want to test your handler with. This should be stored in
tests/messages
such astests/messages/distgit/new_commit.json
.Create a new file in
tests/handlers/
such astests/handlers/distgit.py
.At the top of the new file, add the following license header and imports:
# SPDX-License-Identifier: GPL-3.0+ from __future__ import unicode_literals, absolute_import import json from os import path
Then you can proceed to create your unit test in the same file as such:
from estuary.models.distgit import DistGitCommit from tests import message_dir from estuary_updater.handlers.distgit import DistGitHandler from estuary_updater import config def test_distgit_new_commit(): """Test the dist-git handler when it recieves a new commit message.""" # Load the message to pass to the handler with open(path.join(message_dir, 'distgit', 'new_commit.json'), 'r') as f: msg = json.load(f) # Make sure the handler can handle the message assert DistGitHandler.can_handle(msg) is True # Instantiate the handler handler = DistGitHandler(config) # Run the handler handler.handle(msg) # Verify everything looks good in Neo4j commit = DistGitCommit.nodes.get_or_none(hash_='some_hash_from_the_message') assert commit is not None # Do additional checks here
Handlers¶
Base¶
-
class
estuary_updater.handlers.base.
BaseHandler
(config)[source]¶ An abstract base class for handlers to enforce the API.
-
static
can_handle
(msg)[source]¶ Determine if this handler can handle this message.
Parameters: msg (dict) – a message to be analyzed Returns: a bool based on if the handler can handle this kind of message Return type: bool
-
get_or_create_build
(identifier, original_nvr=None, force_container_label=False)[source]¶ Get a Koji build from Neo4j, or create it if it does not exist in Neo4j.
Parameters: - identifier (str/int) – an NVR (str) or build ID (int), or a dict of info from Koji API
- original_nvr (str) – original_nvr property for the ContainerKojiBuild
- force_container_label (bool) – when true, this skips the check to see if the build is a container and just creates the build with the ContainerKojiBuild label
Return type: KojiBuild
Returns: the Koji Build retrieved or created from Neo4j
-
handle
(msg)[source]¶ Handle a message and update Neo4j if necessary.
Parameters: msg (dict) – a message to be processed
-
is_container_build
(build_info)[source]¶ Check whether a Koji build is a container build.
Parameters: build_info (KojiBuild) – build info from the Koji API Returns: boolean value indicating whether the build is a container build Return type: bool
-
is_module_build
(build_info)[source]¶ Check whether a Koji build is a module build.
Parameters: build_info (KojiBuild) – build info from Koji API Returns: boolean value indicating whether the build is a module build Return type: bool
-
koji_session
¶ Get a cached Koji session but initialize the connection first if needed.
Returns: a Koji session object Return type: koji.ClientSession
-
static
DistGit¶
-
class
estuary_updater.handlers.distgit.
DistGitHandler
(config)[source]¶ A handler for dist-git related messages.
-
static
can_handle
(msg)[source]¶ Determine if this is a dist-git message.
Parameters: msg (dict) – a message to be analyzed Returns: a bool based on if the handler can handle this kind of message Return type: bool
-
commit_handler
(msg)[source]¶ Handle a dist-git commit message and update Neo4j if necessary.
Parameters: msg (dict) – a message to be processed
-
static
Errata¶
-
class
estuary_updater.handlers.errata.
ErrataHandler
(config)[source]¶ A handler for dist-git related messages.
-
advisory_handler
(msg)[source]¶ Handle an Errata tool advisory changes and update Neo4j if necessary.
Parameters: msg (dict) – a message to be processed
-
builds_added_handler
(msg)[source]¶ Handle an Errata tool builds added message and update Neo4j if necessary.
Parameters: msg (dict) – a message to be processed
-
builds_removed_handler
(msg)[source]¶ Handle an Errata tool builds removed message and update Neo4j if necessary.
Parameters: msg (dict) – a message to be processed
-
Freshmaker¶
-
class
estuary_updater.handlers.freshmaker.
FreshmakerHandler
(config)[source]¶ A handler for Freshmaker-related messages.
-
build_state_handler
(msg)[source]¶ Handle a Freshmaker build state changed message and update Neo4j if necessary.
Parameters: msg (dict) – a message to be processed
-
static
can_handle
(msg)[source]¶ Determine if the message is a Freshmaker-related message and can be handled by this handler.
Parameters: msg (dict) – a message to be analyzed Returns: a bool based on if the handler can handle this kind of message Return type: bool
-
create_or_update_build
(build, event_id)[source]¶ Use the Koji Task Result to create or update a ContainerKojiBuild.
Parameters: Returns: the created/updated ContainerKojiBuild or None if it cannot be created
Return type: ContainerKojiBuild or None
-
create_or_update_freshmaker_build
(build, event_id)[source]¶ Create or update a FreshmakerBuild.
Parameters: Returns: the created/updated FreshmakerBuild or None if it cannot be created
Return type: FreshmakerBuild or None
-
Koji¶
-
class
estuary_updater.handlers.koji.
KojiHandler
(config)[source]¶ A handler for Koji related messages.
-
build_handler
(msg)[source]¶ Handle a build state message and update Neo4j if necessary.
Parameters: msg (dict) – a message to be processed
-