Logging

by Dmitry Levichev

Any and all nontrivial systems need logging, and microservices are no exception. Messages in logs help us track running transactions and sort out any problems that may occur. The quality of the information that is stored in logs largely defines how simple or difficult it is to support a system.
The Pip.Services Toolkit contains logging components that can either output messages to the console, or hand them over to specialized services, such as ElasticSearch, AppInsights or CloudWatch.

The logger interface is defined in the Log package of the Components module. This interface contains methods that are common for most logger implementations.

interface ILogger {
  LogLevel Level { get; set; }
   void Log(LogLevel level, String correlationId, Exception error, String message, params Object[] args);
    void Fatal(String correlationId, Exception error, String message, params Object[] args);
    void Error(String correlationId, Exception error, String message, params Object[] args);
    void Warn(String correlationId, String message, params Object[] args);
    void Info(String correlationId, String message, params Object[] args);
    void Debug(String correlationId, String message, params Object[] args);
    void Trace(String correlationId, String message, params Object[] args);
}

The log method writes messages to the log, using the provided log level. The other methods collect messages at various logging levels and can be used in your code to add transparency.
The level property can be used to optimize code. Generating log messages can cost you resources. We can optimize the use of these resources by only generating messages for the log level that is currently set, as is shown in the example below:

if (logger.Level >= LogLevel.Trace) {
message = ... // Costly message preparation
logger.Trace(correlationId, message);
}

One differentiating factor of the Pip.Services Toolkit is the required correlationId parameter. CorrelationId is used in all business methods and is passed in a standardized way when calling microservices. This way, the correlationId is passed along the entire chain of microservice calls, from start to finish, and is included in any and all errors and log messages. This allows us to grasp an understanding of what’s going on, in conditions where information is fragmented and collected from various sources.
To be able to generate quality logs, it’s crucial to know how the various LogLevels should be used. For some reason, most developers don’t consider this to be important, which makes it harder for users, technical support, and for developers themselves to use the system. The main purpose of LogLevel is to filter messages by their importance. When LogLevel is used incorrectly, important messages are bound to be lost, or the opposite can occur, where the output is spammed by various messages, making the search for information a real burden.

None - disable all messages
Fatal - critical errors that lead to partial or complete system failures
Error - user errors that don’t affect the system’s performance
Warning - warnings that technical support should check out
Info - messages relating to important transactions running in the system. These help get an idea of what’s going on in the system under normal conditions.
Debug - detailed information on what’s happening in the system. These messages are used by technical support and are turned on for short periods of time to help debug the system.
Trace - very detailed information on what’s happening in the system. These messages are used only by developers and are almost never turned on in production.

It’s also important to write messages to the log in such a way that they can be understood by people who don’t possess knowledge of the inner workings of the system.

The Pip.Services Toolkit contains a variety of logger implementations:

NullLogger] - Empty logger for debugging (in the Components module)
ConsoleLogger - Logger for writing messages to the console (in the Components module)
CompositeLogger - Virtual logger for collecting and transferring messages to other loggers (in the Components module)
ElasticSearchLogger - Logger for saving messages in ElasticSearch (inthe ElasticSearch module)
FluentdLogger - Logger for transferring messages to Fluentd (in the Fluentd module)
CloudWatchLogger - Logger for collecting messages in AWS CloudWatch (in the AWS module)
AppInsightsLogger - Logger for collecting messages in Azure AppInsights (in the Azure module)

Loggers are usually added to microservices dynamically using a yml configuration:
‍config.yml

. . .
# Console logger
- descriptor: "pip-services:logger:console:default:1.0"
  level: "trace"
# ElasticSearch logger
- descriptor: "pip-services:log:elasticsearch:default:1.0"
source: "test"
connection:
    protocol: "http"
   host: "localhost"
   port: 9200
  options:
    interval: 10000
    max_cache_size: 100
    index: "log"
    daily: true

It’s not rare for multiple loggers to be used simultaneously. One logger can be used to output messages to the console, allowing developers to debug the microservice, while another logger collects messages from all running microservices in a distributed storage, allowing technical support to monitor the system.
To simplify the collection of log messages in situations when the amount of loggers and/or their configurations are bound to change, the CompositeLogger from the Components module is used. The CompositeLogger’s task is to pass along any messages it receives to all of the other loggers included in the container it’s in. Logger linking is performed in the SetReferences method (see the References Recipe).

The CompositeLogger is used in the following way:

class MyComponent {
private _logger: CompositeLogger = new CompositeLogger();
public setReferences(references: IReferences): void {
        this._logger.setReferences(references);
…
          }
public myMethod(String correlationId, ...) {
  this._logger.trace(correlationId, “MyMethod is executed.”);
}
}

Example output from the console logger:

The logger interface is defined in the Log package of the Components module. This interface contains methods that are common for most logger implementations.

public interface ILogger
{
       LogLevel Level { get; set; }
       void Log(LogLevel level, string correlationId, Exception error, string message, params object[] args);
       void Fatal(string correlationId, string message, params object[] args);
       void Fatal(string correlationId, Exception error, string message = null, params object[] args);
       void Error(string correlationId, string message, params object[] args);
       void Error(string correlationId, Exception error, string message = null, params object[] args);
       void Warn(string correlationId, string message, params object[] args);
       void Warn(string correlationId, Exception error, string message = null, params object[] args);
       void Info(string correlationId, string message, params object[] args);
       void Info(string correlationId, Exception error, string message = null, params object[] args);
       void Debug(string correlationId, string message, params object[] args);
       void Debug(string correlationId, Exception error, string message = null, params object[] args);
       void Trace(string correlationId, string message, params object[] args);
       void Trace(string correlationId, Exception error, string message = null, params object[] args);
}

if (logger.Level >= LogLevel.Trace) {
message = ... // Costly message preparation
logger.Trace(correlationId, message);
}

None - disable all messages
Fatal - critical errors that lead to partial or complete system failures
Error - user errors that don’t affect the system’s performance
Warning - warnings that technical support should check out
Info - messages relating to important transactions running in the system. These help get an idea of what’s going on in the system under normal conditions.
Debug - detailed information on what’s happening in the system. These messages are used by technical support and are turned on for short periods of time to help debug the system.
Trace - very detailed information on what’s happening in the system. These messages are used only by developers and are almost never turned on in production.

It’s also important to write messages to the log in such a way that they can be understood by people who don’t possess knowledge of the inner workings of the system.

The Pip.Services Toolkit contains a variety of logger implementations:

NullLogger - Empty logger for debugging (in the Components module)
ConsoleLogger - Logger for writing messages to the console (in the Components module)
CompositeLogger - Virtual logger for collecting and transferring messages to other loggers (in the Components module)
ElasticSearchLogger - Logger for saving messages in ElasticSearch (inthe ElasticSearch module)
FluentdLogger - Logger for transferring messages to Fluentd (in the Fluentd module)
CloudWatchLogger - Logger for collecting messages in AWS CloudWatch (in the AWS module)
AppInsightsLogger - Logger for collecting messages in Azure AppInsights (in the Azure module)

Loggers are usually added to microservices dynamically using a yml configuration:
‍config.yml

. . .
# Console logger
- descriptor: "pip-services:logger:console:default:1.0"
  level: "trace"
# ElasticSearch logger
- descriptor: "pip-services:log:elasticsearch:default:1.0"
source: "test"
connection:
    protocol: "http"
   host: "localhost"
   port: 9200
  options:
    interval: 10000
    max_cache_size: 100
    index: "log"
    daily: true

The CompositeLogger is used in the following way:

private CompositeLogger Log;
Log = new CompositeLogger();
public void SetReferences(IReferences refs){
    Log.SetReferences(refs);
    …
}
public void MyMethod(correlationId) {
   Logger.Trace(correlationId, "Trace message");
    ....
}

Example output from the console logger:

‍

The logger interface is defined in the Log package of the Components module. This interface contains methods that are common for most logger implementations.

class ILogger:
    def get_level(self):
    def set_level(self, level):
    def log(self, level, correlation_id, error, message, *args, **kwargs):
    def fatal(self, correlation_id, error, message, *args, **kwargs):
    def error(self, correlation_id, error, message, *args, **kwargs):
    def warn(self, correlation_id, message, *args, **kwargs):
    def info(self, correlation_id, message, *args, **kwargs):
    def debug(self, correlation_id, message, *args, **kwargs):
    def trace(self, correlation_id, message, *args, **kwargs):

if (logger.Level >= LogLevel.Trace) {
message = ... // Costly message preparation
logger.Trace(correlationId, message);
}

None - disable all messages
Fatal - critical errors that lead to partial or complete system failures
Error - user errors that don’t affect the system’s performance
Warning - warnings that technical support should check out
Info - messages relating to important transactions running in the system. These help get an idea of what’s going on in the system under normal conditions.
Debug - detailed information on what’s happening in the system. These messages are used by technical support and are turned on for short periods of time to help debug the system.
Trace - very detailed information on what’s happening in the system. These messages are used only by developers and are almost never turned on in production.

It’s also important to write messages to the log in such a way that they can be understood by people who don’t possess knowledge of the inner workings of the system.

The Pip.Services Toolkit contains a variety of logger implementations:

NullLogger - Empty logger for debugging (in the Components module)
ConsoleLogger - Logger for writing messages to the console (in the Components module)
CompositeLogger - Virtual logger for collecting and transferring messages to other loggers (in the Components module)
ElasticSearchLogger - Logger for saving messages in ElasticSearch (inthe ElasticSearch module)
FluentdLogger - Logger for transferring messages to Fluentd (in the Fluentd module)
CloudWatchLogger - Logger for collecting messages in AWS CloudWatch (in the AWS module)
AppInsightsLogger - Logger for collecting messages in Azure AppInsights (in the Azure module)

Loggers are usually added to microservices dynamically using a yml configuration:
‍config.yml

. . .
# Console logger
- descriptor: "pip-services:logger:console:default:1.0"
  level: "trace"
# ElasticSearch logger
- descriptor: "pip-services:log:elasticsearch:default:1.0"
source: "test"
connection:
    protocol: "http"
   host: "localhost"
   port: 9200
  options:
    interval: 10000
    max_cache_size: 100
    index: "log"
    daily: true

The CompositeLogger is used in the following way:‍

class MyComponent(IConfigurable, IReferenceable):
    _logger = CompositeLogger()
def configure(self, config):
    self._logger.configure(config)
def set_references(self, references):
    self._logger.set_references(references)

def my_method(self, correlation_id):
self._logger.debug(correlationId, “Called method mycomponent.mymethod”

Example output from the console logger:

The logger interface is defined in the Log package of the Components module. This interface contains methods that are common for most logger implementations.

public interface ILogger {
    LogLevel getLevel();
    void setLevel(LogLevel value);
    void log(LogLevel level, String correlationId, Exception error, String message, Object... args);
    void fatal(String correlationId, String message, Object... args);
    void fatal(String correlationId, Exception error);
    void fatal(String correlationId, Exception error, String message, Object... args);
    void error(String correlationId, String message, Object... args);
    void error(String correlationId, Exception error);
    void error(String correlationId, Exception error, String message, Object... args);
    void warn(String correlationId, String message, Object... args);
    void info(String correlationId, String message, Object... args);
    void debug(String correlationId, String message, Object... args);
    void trace(String correlationId, String message, Object... args);
}

if (logger.Level >= LogLevel.Trace) {
message = ... // Costly message preparation
logger.Trace(correlationId, message);
}

None - disable all messages
Fatal - critical errors that lead to partial or complete system failures
Error - user errors that don’t affect the system’s performance
Warning - warnings that technical support should check out
Info - messages relating to important transactions running in the system. These help get an idea of what’s going on in the system under normal conditions.
Debug - detailed information on what’s happening in the system. These messages are used by technical support and are turned on for short periods of time to help debug the system.
Trace - very detailed information on what’s happening in the system. These messages are used only by developers and are almost never turned on in production.

It’s also important to write messages to the log in such a way that they can be understood by people who don’t possess knowledge of the inner workings of the system.

The Pip.Services Toolkit contains a variety of logger implementations:

NullLogger - Empty logger for debugging (in the Components module)
ConsoleLogger - Logger for writing messages to the console (in the Components module)
CompositeLogger - Virtual logger for collecting and transferring messages to other loggers (in the Components module)
ElasticSearchLogger - Logger for saving messages in ElasticSearch (inthe ElasticSearch module)
FluentdLogger - Logger for transferring messages to Fluentd (in the Fluentd module)
CloudWatchLogger - Logger for collecting messages in AWS CloudWatch (in the AWS module)
AppInsightsLogger - Logger for collecting messages in Azure AppInsights (in the Azure module)

Loggers are usually added to microservices dynamically using a yml configuration:
‍config.yml

. . .
# Console logger
- descriptor: "pip-services:logger:console:default:1.0"
  level: "trace"
# ElasticSearch logger
- descriptor: "pip-services:log:elasticsearch:default:1.0"
source: "test"
connection:
    protocol: "http"
   host: "localhost"
   port: 9200
  options:
    interval: 10000
    max_cache_size: 100
    index: "log"
    daily: true

The CompositeLogger is used in the following way:

CompositeLogger _log;
void setReferences(IReferences references) {
    logger.setReferences(references);
    …
}
void myMethod(String correlationId, ...){
  _log.trace("123", “MyMethod is executed.”);
...
}

Example output from the console logger:

‍

The logger interface is defined in the Log package of the Components module. This interface contains methods that are common for most logger implementations.‍

type ILogger interface {
  Level() int
  SetLevel(value int)
  Log(level int, correlationId string, err error, message string, args ...interface{})
  Fatal(correlationId string, err error, message string, args ...interface{})
  Error(correlationId string, err error, message string, args ...interface{})
  Warn(correlationId string, message string, args ...interface{})
  Info(correlationId string, message string, args ...interface{})
  Debug(correlationId string, message string, args ...interface{})
  Trace(correlationId string, message string, args ...interface{})
}

if (logger.Level >= LogLevel.Trace) {
message = ... // Costly message preparation
logger.Trace(correlationId, message);
}

None - disable all messages
Fatal - critical errors that lead to partial or complete system failures
Error - user errors that don’t affect the system’s performance
Warning - warnings that technical support should check out
Info - messages relating to important transactions running in the system. These help get an idea of what’s going on in the system under normal conditions.
Debug - detailed information on what’s happening in the system. These messages are used by technical support and are turned on for short periods of time to help debug the system.
Trace - very detailed information on what’s happening in the system. These messages are used only by developers and are almost never turned on in production.

It’s also important to write messages to the log in such a way that they can be understood by people who don’t possess knowledge of the inner workings of the system.

The Pip.Services Toolkit contains a variety of logger implementations:

NullLogger - Empty logger for debugging (in the Components module)
ConsoleLogger - Logger for writing messages to the console (in the Components module)
CompositeLogger - Virtual logger for collecting and transferring messages to other loggers (in the Components module)
ElasticSearchLogger - Logger for saving messages in ElasticSearch (inthe ElasticSearch module)
FluentdLogger - Logger for transferring messages to Fluentd (in the Fluentd module)
CloudWatchLogger - Logger for collecting messages in AWS CloudWatch (in the AWS module)
AppInsightsLogger - Logger for collecting messages in Azure AppInsights (in the Azure module)

Loggers are usually added to microservices dynamically using a yml configuration:
‍config.yml

. . .
# Console logger
- descriptor: "pip-services:logger:console:default:1.0"
  level: "trace"
# ElasticSearch logger
- descriptor: "pip-services:log:elasticsearch:default:1.0"
source: "test"
connection:
    protocol: "http"
   host: "localhost"
   port: 9200
  options:
    interval: 10000
    max_cache_size: 100
    index: "log"
    daily: true

The CompositeLogger is used in the following way:‍

Logger clog.CompositeLogger
func (c ObjectController) SetReferences(references crefer.IReferences) {
c.Logger.SetReferences(references)
...
}

func (c *ObjectController) MyMethod(CorrelationId string) {
c.Logger.Trace(CorrelationId, “MyMethod is executed.”)
...
}

Example output from the console logger:

‍

The logger interface is defined in the Log package of the Components module. This interface contains methods that are common for most logger implementations.

abstract class ILogger {
  LogLevel getLevel();
  void setLevel(LogLevel value);
  void log( LogLevel level, String correlationId, Exception error, String message, [List args]);
  void fatal(String correlationId, Exception error, String message, [List args]);
  void error(String correlationId, Exception error, String message, [List args]);
  void warn(String correlationId, String message, [List args]);
  void info(String correlationId, String message, [List args]);
  void debug(String correlationId, String message, [List args]);
  void trace(String correlationId, String message, [List args]);
}

if (logger.Level >= LogLevel.Trace) {
message = ... // Costly message preparation
logger.Trace(correlationId, message);
}

None - disable all messages
Fatal - critical errors that lead to partial or complete system failures
Error - user errors that don’t affect the system’s performance
Warning - warnings that technical support should check out
Info - messages relating to important transactions running in the system. These help get an idea of what’s going on in the system under normal conditions.
Debug - detailed information on what’s happening in the system. These messages are used by technical support and are turned on for short periods of time to help debug the system.
Trace - very detailed information on what’s happening in the system. These messages are used only by developers and are almost never turned on in production.

It’s also important to write messages to the log in such a way that they can be understood by people who don’t possess knowledge of the inner workings of the system.

The Pip.Services Toolkit contains a variety of logger implementations:

NullLogger - Empty logger for debugging (in the Components module)
ConsoleLogger - Logger for writing messages to the console (in the Components module)
CompositeLogger - Virtual logger for collecting and transferring messages to other loggers (in the Components module)
ElasticSearchLogger - Logger for saving messages in ElasticSearch (inthe ElasticSearch module)
FluentdLogger - Logger for transferring messages to Fluentd (in the Fluentd module)
CloudWatchLogger - Logger for collecting messages in AWS CloudWatch (in the AWS module)
AppInsightsLogger - Logger for collecting messages in Azure AppInsights (in the Azure module)

Loggers are usually added to microservices dynamically using a yml configuration:
‍config.yml

. . .
# Console logger
- descriptor: "pip-services:logger:console:default:1.0"
  level: "trace"
# ElasticSearch logger
- descriptor: "pip-services:log:elasticsearch:default:1.0"
source: "test"
connection:
    protocol: "http"
   host: "localhost"
   port: 9200
  options:
    interval: 10000
    max_cache_size: 100
    index: "log"
    daily: true

The CompositeLogger is used in the following way:

var logger = CompositeLogger();
@override
  void setReferences(IReferences references) {
    logger.setReferences(references);
    …
}
void myMethod(String correlationId, ...){
  logger.trace(correlationId, “MyMethod is executed.”);
  ...
}

Example output from the console logger:

‍

Logging

by Dmitry Levichev

if (logger.Level >= LogLevel.Trace) { message = ... // Costly message preparation logger.Trace(correlationId, message);}

class MyComponent {private _logger: CompositeLogger = new CompositeLogger(); public setReferences(references: IReferences): void { this._logger.setReferences(references); … }public myMethod(String correlationId, ...) { this._logger.trace(correlationId, “MyMethod is executed.”);}}

if (logger.Level >= LogLevel.Trace) { message = ... // Costly message preparation logger.Trace(correlationId, message);}

private CompositeLogger Log;Log = new CompositeLogger();public void SetReferences(IReferences refs){ Log.SetReferences(refs); …}public void MyMethod(correlationId) { Logger.Trace(correlationId, "Trace message"); ....}

if (logger.Level >= LogLevel.Trace) { message = ... // Costly message preparation logger.Trace(correlationId, message);}

class MyComponent(IConfigurable, IReferenceable): _logger = CompositeLogger()def configure(self, config): self._logger.configure(config)def set_references(self, references): self._logger.set_references(references)

def my_method(self, correlation_id): self._logger.debug(correlationId, “Called method mycomponent.mymethod”

if (logger.Level >= LogLevel.Trace) { message = ... // Costly message preparation logger.Trace(correlationId, message);}

CompositeLogger _log;void setReferences(IReferences references) { logger.setReferences(references); …}void myMethod(String correlationId, ...){ _log.trace("123", “MyMethod is executed.”);...}

if (logger.Level >= LogLevel.Trace) { message = ... // Costly message preparation logger.Trace(correlationId, message);}

Logger *clog.CompositeLoggerfunc (c *ObjectController) SetReferences(references crefer.IReferences) { c.Logger.SetReferences(references) ...}

func (c *ObjectController) MyMethod(CorrelationId string) { c.Logger.Trace(CorrelationId, “MyMethod is executed.”) ...}

if (logger.Level >= LogLevel.Trace) { message = ... // Costly message preparation logger.Trace(correlationId, message);}

var logger = CompositeLogger();@override void setReferences(IReferences references) { logger.setReferences(references); …}void myMethod(String correlationId, ...){ logger.trace(correlationId, “MyMethod is executed.”); ...}

if (logger.Level >= LogLevel.Trace) {
message = ... // Costly message preparation
logger.Trace(correlationId, message);
}

class MyComponent {
private _logger: CompositeLogger = new CompositeLogger();
public setReferences(references: IReferences): void {
this._logger.setReferences(references);
…
}
public myMethod(String correlationId, ...) {
this._logger.trace(correlationId, “MyMethod is executed.”);
}
}

if (logger.Level >= LogLevel.Trace) {
message = ... // Costly message preparation
logger.Trace(correlationId, message);
}

private CompositeLogger Log;
Log = new CompositeLogger();
public void SetReferences(IReferences refs){
Log.SetReferences(refs);
…
}
public void MyMethod(correlationId) {
Logger.Trace(correlationId, "Trace message");
....
}

if (logger.Level >= LogLevel.Trace) {
message = ... // Costly message preparation
logger.Trace(correlationId, message);
}

class MyComponent(IConfigurable, IReferenceable):
_logger = CompositeLogger()
def configure(self, config):
self._logger.configure(config)
def set_references(self, references):
self._logger.set_references(references)

def my_method(self, correlation_id):
self._logger.debug(correlationId, “Called method mycomponent.mymethod”

if (logger.Level >= LogLevel.Trace) {
message = ... // Costly message preparation
logger.Trace(correlationId, message);
}

CompositeLogger _log;
void setReferences(IReferences references) {
logger.setReferences(references);
…
}
void myMethod(String correlationId, ...){
_log.trace("123", “MyMethod is executed.”);
...
}

if (logger.Level >= LogLevel.Trace) {
message = ... // Costly message preparation
logger.Trace(correlationId, message);
}

Logger clog.CompositeLogger
func (c ObjectController) SetReferences(references crefer.IReferences) {
c.Logger.SetReferences(references)
...
}

func (c *ObjectController) MyMethod(CorrelationId string) {
c.Logger.Trace(CorrelationId, “MyMethod is executed.”)
...
}

if (logger.Level >= LogLevel.Trace) {
message = ... // Costly message preparation
logger.Trace(correlationId, message);
}

var logger = CompositeLogger();
@override
void setReferences(IReferences references) {
logger.setReferences(references);
…
}
void myMethod(String correlationId, ...){
logger.trace(correlationId, “MyMethod is executed.”);
...
}