In my next blog post I will talk about complexity of freeBSD VFS using ZFS. Specifically, structured as a series of causally-related actions. The next step is making sure log messages are consistent: objects that are used in different log messages should be referred to consistently, messages should be organized in a consistent manner to ease searching, etc.. The way the Eliot logging library does this is by providing a type system for messages, built out of “fields” that know how to serialize arbitrary Python objects. For example, let’s declare a logging action that describes a state machine transition. The start of the action will include the identity of the state machine, its current state, and the input. The end of the action, if successful, will include the new state and some outputs. from eliot import Field, ActionType # A Field that knows how to serialize an object to a loggable format: FSM_IDENTIFIER = Field(u”fsm_identifier”, lambda fsm: fsm.identifier(), u”An unique identifier for the FSM to which the event pertains.”) # Some fields that merely expect to receive inputs of specified types: FSM_STATE = Field.forTypes( u”fsm_state”, [unicode], u”The state of the FSM prior to the transition.”) FSM_INPUT = Field.forTypes( u”fsm_input”, [unicode], u”The string representation of the input symbol delivered to the FSM.”) FSM_NEXT_STATE = Field.forTypes( u”fsm_next_state”, [unicode], u”The string representation of the state of the FSM after the transition.”) FSM_OUTPUT = Field.forTypes( u”fsm_output”, [list], # of unicode u”A list of the string representations of the outputs produced by the ” u”transition.”) # The definition of an action: LOG_FSM_TRANSITION = ActionType( # The name of the action: u”fsm:transition”, # Fields included in the start message of the action: [FSM_IDENTIFIER, FSM_STATE, FSM_INPUT], # Fields included in the successful end message of the action: [FSM_NEXT_STATE, FSM_OUTPUT], # Fields (beyond built-in exception and reason) included in the failure end # message of the action: [], # Description of the action: u”A finite state machine received an input made a transition.”) We can now use this to log actions in our state machine implementation: from eliot import Logger class FiniteStateMachine(object): logger = Logger() def __init__(self, name, state, transitions, handler): self.name = name self.state = state self.transitions = transitions self.handler = handler def identifier(self): return self.name def input(self, what): with LOG_FSM_TRANSITION(self.logger, fsm_identifier=self, fsm_state=self.state, fsm_input=what) as action: # Look up the state machine transition: outputs, next_state = self.transitions[self.state][what] # Tell the action what fields to put in the success message: action.addSuccessBindings(fsm_next_state=next_state, fsm_output=outputs) # Handler’s logging will be in the context of the # LOG_FSM_TRANSITION action: for output in outputs: self.handler(output) self.state = next_state What benefits do we get from having an explicit type and fields? In my next post I’ll talk about unit testing, ensuring your logging code is written correctly and actually being run. Meanwhile why not read more about HybridCluster’s technology underlying our cloud platform for web hosting, the reason this logging system is being written.
I am happy to announce that Eliot, a logging library for Python, is now available as an open source project. In previous posts (now part of the documentation) I talked about the motivation behind Eliot: logging as storytelling. Log messages in Eliot are a forest of nested actions. Actions start and eventually finish, successfully or not. Log messages thus tell a story: what happened and what caused it. Here’s what your logs might look like before using Eliot: Going to validate http://example.com/index.html. Started download attempted. Download succeeded! Missing <title> element in “/html/body”. Bad HTML entity in “/html/body/p[2]”. 2 validation errors found! After switching to Eliot you’ll get a tree of messages with both message contents and causal relationships encoded in a structured format: {“action_type”: “validate_page”, “action_status”: “started”, “url”: “http://example.com/index.html”} {“action_type”: “download”, “action_status”: “started”} {“action_type”: “download”, “action_status”: “succeeded”} {“action_type”: “validate_html”, “action_status”: “started”} {“message_type”: “validation_error”, “error_type”: “missing_title”, “xpath”: “/html/head”} {“message_type”: “validation_error”, “error_type”: “bad_entity”, “xpath”: “/html/body/p[2]”} {“action_type”: “validate_html”, “action_status”: “failed”, “exception”: “validator.ValidationFailed”} {“action_type”: “validate_page”, “action_status”: “failed”, “exception”: “validator.ValidationFailed”} To install: $ pip install eliot Documentation can be found on Read The Docs. Bugs and feature requests should be filed at the project Github page. More to read: Containers and distributed storage are the future
