Memory persistence

by Artyom Grishchenko

       The Pip.Services Toolkit offers a few abstract implementations for developing persistent components. One of them is the MemoryPersistence, which stores all of its data in memory. Its usefulness is limited in production, but very handy in unit tests. This persistence allows us to cut dependencies on external persistent storages and makes tests easy to set up and lighting fast!
      The most basic implementation is the MemoryPersistence class defined in the Data module. It is only capable of storing a collection of objects, opening, and closing. It does not provide any data access methods.
    The implementation we will be working with is called IdentifiableMemoryPersistence. It stores and processes data objects that have a unique ID field and implement the IIdentifiable interface defined in the Commons module.

export interface IIdentifiable<K> {
id: K;
}

IdentifiableMemoryPersistence implements a number of CRUD methods:

export class IdentifiableMemoryPersistence<T extends IIdentifiable<K>, K> extends MemoryPersistence<T> implements IConfigurable {
protected _items: any[];
protected getPageByFilter(correlationId: string, filter: any,
paging: PagingParams, sort: any, select: any,
callback: (err: any, page: DataPage<T>) => void): void;
protected getCountByFilter(correlationId: string, filter: any,
callback: (err: any, count: number) => void): void;
protected getListByFilter(correlationId: string, filter: any, sort: any, select: any,
callback: (err: any, items: T[]) => void): void;
public getListByIds(correlationId: string, ids: K[],
callback: (err: any, items: T[]) => void): void;
protected getOneRandom(correlationId: string, filter: any, callback: (err: any, item: T) => void): void;
public getOneById(correlationId: string, id: K, callback: (err: any, item: T) => void): void;
public create(correlationId: string, item: T, callback?: (err: any, item: T) => void): void;
public set(correlationId: string, item: T, callback?: (err: any, item: T) => void): void;
public update(correlationId: string, item: T, callback?: (err: any, item: T) => void): void;
public updatePartially(correlationId: string, id: K, data: AnyValueMap,
callback?: (err: any, item: T) => void): void;
public deleteById(correlationId: string, id: K, callback?: (err: any, item: T) => void): void;
protected deleteByFilter(correlationId: string, filter: any, callback?: (err: any) => void): void;
public deleteByIds(correlationId: string, ids: K[], callback?: (err: any) => void): void;
}

In most scenarios, child classes only need to override the GetPageByFilter(), GetListByFilter(), or DeleteByFilter() operations using a custom filter function. All other operations can be used right out of the box. Developers can implement custom methods by accessing stored data objects via the this._items property and complete transactions by calling the Save() method. See the Data module’s API documentation for more details.

Filtering

Persistent components in the Pip.Services Toolkit use a number of data patterns. IdentifiedMemoryPersistence, for example, supports Filtering. This pattern allows clients to use a FilterParams object to describe a subset of data as key-value pairs. These FilterParams can then be used for retrieving data in accordance with certain search criteria (see the Commons module).

let filter = FilterParams.fromTuples(
‘name’, 'ABC'
)
persistence.getPageFilter(null, filter, null, (page, err) => {
...
});

In the persistence component, the developer is responsible for parsing the FilterParams and passing a filter function to the persistent methods of the base class.

   private composeFilter(filter: FilterParams): any {
filter = filter || new FilterParams();
let name = filter.getAsNullableString("name");
return (item) => {
if (name != null && item.name != name)
return false;
return true;
};
}
  public getPageByFilter(correlationId: string, filter: FilterParams, paging: PagingParams,
callback: (err: any, page: DataPage<MyData>) => void): void {
super.getPageByFilter(correlationId, this.composeFilter(filter), paging, null, null, callback);
  }

Paging

Another common data pattern is Paging. It is used to retrieve large datasets in chunks through multiple calls to the storage. To do this, a client specifies a set of PagingParams, which include the starting position and the number of objects to return. Clients can also request the total number of items in the dataset using PagingParams, but this parameter is optional. The service returns a subset of the data as a DataPage object.

//skip = 25, take = 50, total = false
let paging = new PagingParams(25, 50, false);
persistence.getPageByFilter(null, null, paging, (page, err) => {|
…
});

Custom Persistence Methods

As mentioned above, developers can also implement custom persistent methods. Inside those methods, they can access data objects via the _items property. When stored data is modified, developers must finish the transaction by calling the base class’s Save() method.
Below is an example of a custom persistent method.

public getOneByName(correlationId: string, name: string,
callback: (err: any, item: MyData) => void): void {
let item = _.find(this._items, (item) => item.name == name);
if (item != null) this._logger.trace(correlationId, "Found by %s", name);
else this._logger.trace(correlationId, "Cannot find by %s", name);
callback(null, item);
}

When we put everything together, we get the following component:

class MyMemoryPersistence extends IdentifiableMemoryPersistence<MyData, string> {
private composeFilter(filter: FilterParams): any {
filter = filter || new FilterParams();
let name = filter.getAsNullableString("name");
return (item) => {
if (name != null && item.name != name)
return false;
return true;
};
}
public getPageByFilter(correlationId: string, filter: FilterParams, paging: PagingParams,
callback: (err: any, page: DataPage<MyData>) => void): void {
super.getPageByFilter(correlationId, this.composeFilter(filter), paging, null, null, callback);
}
public getOneByName(correlationId: string, name: string,
callback: (err: any, item: MyData) => void): void {
let item = _.find(this._items, (item) => item.name == name);
if (item != null) this._logger.trace(correlationId, "Found by %s", name);
else this._logger.trace(correlationId, "Cannot find by %s", name);
callback(null, item);
}
}

A demonstration of how we can use our custom memory persistence is presented below:

public useMemoryPersistence(done) {
async.series([
// Create items
(callback) => {
let persistence = new MyMemoryPersistence();
persistence.create("123", { id: "1", name: "ABC" }, null);
},
// Filter by name
(callback) => {
persistence.getPageByFilter(
null,
FilterParams.fromTuples(
"name", "ABC"
),
new PagingParams(0, 100, false),
(err, page) => {
callback();
}
)
},
], done);
}
}

FileMemoryPersistence

The memory persistence component actually has one more trick up its sleeve: it can easily be extended to create a FileMemoryPersistence. The only thing you’ll need to add is the assignment of a PersisterObject in the FileMemoryPersistence’s constructor. The File persistence can be used for certain system test scenarios.

import { JsonFilePersister } from 'pip-services3-data-node';
import { BeaconV1 } from '../data/version1/BeaconV1';
import { MyMemoryPersistence } from './MyMemoryPersistence';
import { ConfigParams } from 'pip-services3-commons-node';
export class MyFilePersistence extends MyMemoryPersistence {
protected _persister: JsonFilePersister<BeaconV1>;
constructor(path?: string) {
super();
this._persister = new JsonFilePersister<BeaconV1>(path);
this._loader = this._persister;
this._saver = this._persister;
}
public configure(config: ConfigParams) {
super.configure(config);
this._persister.configure(config);
}
}

public interface IIdentifiable<T>
{
T Id { get; set; }
}

‍IdentifiableMemoryPersistence implements a number of CRUD methods:

‍

Filtering

FilterParams.fromTuples("name", "ABC")

In the persistence component, the developer is responsible for parsing the FilterParams and passing a filter function to the persistent methods of the base class.

Paging

//skip = 25, take = 50, total = false
new PagingParams(25, 50, false);

Custom Persistence Methods

public async Task<MyData> GetOneByNameAsync(string correlationId, string name)
{
MyData item = null;
lock (_lock)
{
item = _items.Find((x) => { return x.Name == name; });
}
if (item != null)
_logger.Trace(correlationId, "Found by Name {0}", name);
else
_logger.Trace(correlationId, "Cannot find by Id {0}", name);
return await Task.FromResult(item);
}

When we put everything together, we get the following component:

class MyMemoryPersistence: IdentifiableMemoryPersistence<MyData, string>
{
public MyMemoryPersistence()
{
        _maxPageSize = 1000;
}
    private List<Func<MyData, bool>> ComposeFilter(FilterParams filter)
{
filter = filter != null ? filter : new FilterParams();
String name = filter.getAsNullableString("name");
return List<Func<MyData, bool>>() {
     (item) => {
         if (name != null && item.name != name)
        return false;
         return true;
    }};
}
public DataPage<MyData> GetPageByFilter(string correlationId, FilterParams filter, PagingParams paging)
{
base.GetPageByFilter(correlationId, this.composeFilter(filter), paging, null, null);
}
public async Task<MyData> GetOneByNameAsync(string correlationId, string name)
{
MyData item = null;
lock (_lock)
{
item = _items.Find((x) => { return x.Name == name; });
}
if (item != null)
_logger.Trace(correlationId, "Found by name {0}", name);
else
_logger.Trace(correlationId, "Cannot find by name {0}", name);
return await Task.FromResult(item);
}
}

A demonstration of how we can use our custom memory persistence is presented below:

public void UseMemoryPersistenceAsync()
{
   // arrange
    var persistence = new MyMemoryPersistence();
    var item = persistence.Create("123", new MyData("1", "ABC"));
// filter by id
var filter = FilterParams.FromTuples(
"name", "ABC"
);
// act
    var result = persistence.GetPageByFilter(null, filter,new PagingParams(0, 100, false));
    //clean
    persistence.DeleteById("123", "1");
}

FileMemoryPersistence

using My.Data.Version1;
using PipServices3.Commons.Config;
using PipServices3.Data.Persistence;
namespace My.Persistence
{
public class MyFilePersistence : MyMemoryPersistence
{
protected JsonFilePersister<BeaconV1> _persister;
public MyFilePersistence()
{
_persister = new JsonFilePersister<BeaconV1>();
_loader = _persister;
_saver = _persister;
}
public override void Configure(ConfigParams config)
{
base.Configure(config);
_persister.Configure(config);
}
}
}

abstract class IIdentifiable<K> {
K id;
}

IdentifiableMemoryPersistence implements a number of CRUD methods:

class IdentifiableMemoryPersistence<T extends IIdentifiable<K>, K>
extends MemoryPersistence<T>
implements IConfigurable, IWriter<T, K>, IGetter<T, K>, ISetter<T> {
‍
Future<DataPage<T>> getPageByFilterEx(String correlationId, Function filter,
PagingParams paging, Function sort);
Future<List<T>> getListByFilterEx(String correlationId, filter, sort, select);
Future<List<T>> getListByIds(String correlationId, List<K> ids);
Future<T> getOneRandom(String correlationId, filter);
Future<T> getOneById(String correlationId, K id);
Future<T> create(String correlationId, T item);
Future<T> set(String correlationId, T item);
Future<T> update(String correlationId, T item);
Future<T> updatePartially(String correlationId, K id, AnyValueMap data);
Future<T> deleteById(String correlationId, K id);
Future deleteByFilterEx(String correlationId, filter);
Future deleteByIds(String correlationId, List<K> ids);
Future<int> getCountByFilterEx(String correlationId, filter);
}

Filtering

FilterParams.fromTuples(['name', 'ABC'])

In the persistence component, the developer is responsible for parsing the FilterParams and passing a filter function to the persistent methods of the base class.

   _composeFilter(FilterParams filter) {
filter = filter ?? FilterParams();
var name = filter.getAsNullableString("name");
return (item) {
if (name != null && item.name != name)
return false;
return true;
};
}
  Future<DataPage<MyData>> getPageByFilter(String correlationId, FilterParams filter, PagingParams paging) {
return super.getPageByFilterEx(correlationId, this.composeFilter(filter), paging, null, null);
  }

Paging

//skip = 25, take = 50, total = false
PagingParams(25, 50, false);

Custom Persistence Methods

Future<MyData> getOneByName(String correlationId, String name) async {
var item = items.firstWhere((item) => item.name == name);
if (item != null) {
logger.trace(correlationId, 'Found by %s', [name]);
} else {
logger.trace(correlationId, 'Cannot find by %s', [name]);
}
return item;
}
}

When we put everything together, we get the following component:

class MyMemoryPersistence extends IdentifiableMemoryPersistence<MyData, string> {
dynamic _composeFilter(FilterParams filter) {
filter = filter ?? FilterParams();
var name = filter.getAsNullableString("name");
return (item) {
if (name != null && item.name != name)
return false;
return true;
};
}
@override
Future<DataPage<MyData>> getPageByFilter(String correlationId, FilterParams filter, PagingParams paging){
return super.getPageByFilter(correlationId, composeFilter(filter), paging, null, null);
}
@override
Future<MyData> getOneByName(String correlationId, String name) async {
var item = items.firstWhere((item) => item.name == name);
if (item != null) {
logger.trace(correlationId, 'Found by %s', [name]);
} else {
logger.trace(correlationId, 'Cannot find by %s', [name]);
}
return item;
}
}

A demonstration of how we can use our custom memory persistence is presented below:

void UseMemoryPersistence() async {
// Create items
var persistence = MyMemoryPersistence();
// Filter by name
var page = await persistence.getPageByFilter(
null, FilterParams.fromTuples(['name', 'ABC']), PagingParams());
//Clean
item = persistence.deleteById('123', '1');
}

FileMemoryPersistence

import 'package:pip_services3_data/pip_services3_data.dart';
import 'package:pip_services3_commons/pip_services3_commons.dart';
import '../data/version1/BeaconV1.dart';
import './MyMemoryPersistence.dart';
class MyFilePersistence extends MyMemoryPersistence {
  JsonFilePersister<BeaconV1> persister;
  MyFilePersistence([String path]) : super() {
    persister = JsonFilePersister<BeaconV1>(path);
    loader = persister;
    saver = persister;
  }
  @override
  void configure(ConfigParams config) {
    super.configure(config);
    persister.configure(config);
  }
}

Memory persistence

by Artyom Grishchenko

export interface IIdentifiable<K> { id: K;}

Filtering

let filter = FilterParams.fromTuples( ‘name’, 'ABC' )persistence.getPageFilter(null, filter, null, (page, err) => { ...});

Paging

//skip = 25, take = 50, total = falselet paging = new PagingParams(25, 50, false);persistence.getPageByFilter(null, null, paging, (page, err) => {|…});

Custom Persistence Methods

FileMemoryPersistence

public interface IIdentifiable<T> { T Id { get; set; } }

Filtering

FilterParams.fromTuples("name", "ABC")

Paging

//skip = 25, take = 50, total = falsenew PagingParams(25, 50, false);

Custom Persistence Methods

FileMemoryPersistence

abstract class IIdentifiable<K> { K id;}

Filtering

FilterParams.fromTuples(['name', 'ABC'])

Paging

//skip = 25, take = 50, total = falsePagingParams(25, 50, false);

Custom Persistence Methods

Future<MyData> getOneByName(String correlationId, String name) async { var item = items.firstWhere((item) => item.name == name); if (item != null) { logger.trace(correlationId, 'Found by %s', [name]); } else { logger.trace(correlationId, 'Cannot find by %s', [name]); } return item; }}

void UseMemoryPersistence() async { // Create items var persistence = MyMemoryPersistence(); // Filter by name var page = await persistence.getPageByFilter( null, FilterParams.fromTuples(['name', 'ABC']), PagingParams()); //Clean item = persistence.deleteById('123', '1');}

FileMemoryPersistence

export interface IIdentifiable<K> {
id: K;
}

let filter = FilterParams.fromTuples(
‘name’, 'ABC'
)
persistence.getPageFilter(null, filter, null, (page, err) => {
...
});

//skip = 25, take = 50, total = false
let paging = new PagingParams(25, 50, false);
persistence.getPageByFilter(null, null, paging, (page, err) => {|
…
});

public interface IIdentifiable<T>
{
T Id { get; set; }
}

//skip = 25, take = 50, total = false
new PagingParams(25, 50, false);

abstract class IIdentifiable<K> {
K id;
}

//skip = 25, take = 50, total = false
PagingParams(25, 50, false);

Future<MyData> getOneByName(String correlationId, String name) async {
var item = items.firstWhere((item) => item.name == name);
if (item != null) {
logger.trace(correlationId, 'Found by %s', [name]);
} else {
logger.trace(correlationId, 'Cannot find by %s', [name]);
}
return item;
}
}

void UseMemoryPersistence() async {
// Create items
var persistence = MyMemoryPersistence();
// Filter by name
var page = await persistence.getPageByFilter(
null, FilterParams.fromTuples(['name', 'ABC']), PagingParams());
//Clean
item = persistence.deleteById('123', '1');
}