Revision History
1. Introduction
1.1. Purpose and Scope
1.2. General Overview
2. Technical / MAPI Considerations
2.1. MAPI objects
2.2. MAPI tables
2.3. MAPI properties and mapping
2.4. Named properties vs known properties
3. MAPIStore architecture
3.1. INTERFACE layer
3.2. PROCESSING layer
3.3. BACKENDS layer
3.4. Relationship to OpenChange Dispatcher database
3.5. Object mapping
4. MAPIStore API
4.1. Initialization
4.2. Backend contexts
5. FSOCPF backend
5.1. Definition
5.2. Namespace and Attributes
5.3. Overview
5.4. Documentation and References
Date Revision Number Author Revision Content 21/05/10 0.3 Julien Kerihuel Add API documentation for initialization, backend connection contexts and add programming samples 21/05/10 0.2 Julien Kerihuel Merge initial Wiki document and add FSOCPF section 20/05/10 0.1 Julien Kerihuel Draft document.
MAPIStore is the SAL component of OpenChange server. SAL stands for Storage Abstraction Layer. It is the component used by OpenChange Server to push/get information (messages, folders) to/from storage backends. The following document intends to describe the overall/theoretical SAL behavior and constraints we need to consider when dealing with MAPI/EMSMDB. It also describes the semantics and inner working of its storage backends.
The main objective of mapistore is to provide an interface layer with a common set of atomic functions (operations) used to trigger and dispatch data and commands to the appropriate backend. MAPIStore relies on a backend mechanism specifically designed to transparently handle some of the MAPI semantics required by any Exchange compatible server.
The initial idea was to provide to OpenChange a highly customizable storage backend mechanism which would fit in any situation and any environments. One of the greatest limitation we have found with existing groupware is the storage layer which is generally limited to a single solution, service or format and is neither scalable nor modifiable when user requirements evolve upon time.
MAPIStore solves this problem and go beyond classical limitations. It is not a revolutionary concept, but the way openchange uses it makes the whole difference and offer administrators an innovative way to customize storage.
MAPIStore allows you to:
use a different backend for any top-folder
transparently move/copy data across backends
develop new backends quickly
access all the different backends through an unique API
For example (assuming all associated backends were developed) a user could have the following storage organization for his mailbox:
Mails stored using an IMAP backend (Cyrus-IMAP or dovecot)
Calendar items stored in CalDAV or pushed in Google calendar
Sent emails and archives/backup stored in a compression backend
Tasks stored in a MySQL database
Notes stored on the filesystem
If the user is not satisfied with one of the backend's performance, they would just have to use an administration tool, change the backend, wait for the replication, synchronization to finish and there data will be available from the new backend.
Information can be completely decentralized, stored on one of several servers and still be accessible transparently from OpenChange server.
An object is a physical (message, folder) or temporary (table) but generic entity which can be represented as a 2 columns fixed array with n rows, where each row contains a property of that entity. One column repesents the property tags (names), and the other represents the property value (or values).
From a MAPI perspective (network layer), opening an object means:
opening either a private mailbox or public folder store. These are referenced by EssDN and not IDs
opening a message given its PR_MID (Message identifier)
opening a folder/container given its PR_FID (Folder identifier)
Another category of MAPI objects are tables (also known as views) created with MAPI ROPs such as GetContentsTable, GetHierarchyTable, GetAttachmentTable or GetRulesTable. Views are temporary representation of the elements that a container holds at a specific moment. It can be represented as a list of n rows (elements) with n columns (property values):
GetContentsTables creates a view listing all messages available within a container
GetHierarchyTable creates a view listing all containers within a container
GetAttachmentTable creates a view listing all the attachment of a message
GetRulesTable creates a view listing all permissions associated to a container
Tables are customized through the SetColumns MAPI ROP which will define the property identifiers to be retrieved. The QueryRows MAPI ROP can then be called to sequentially retrieve rows and associated property values. A table is similar to a MAPI object except it is virtual, created on demand to represent a list of objects rather than a unique object.
There is a large set of fixed and known MAPI properties available. If appropriate backends are developed, there can be a 1-1 mapping between MAPI properties and backend properties for some of them. This mapping may even be enough for common purposes. However there will still be a set of MAPI properties which won't fit in this process.
There are different way to workaround this issue. For example, Kolab is using a Cyrus/IMAP backend and associate/store MAPI properties to the message using ANNOTATE-MORE within a XML file format.
OpenChange provides similar mechanism with its OCPF file format.
OpenChange server needs to support named properties. An initial set of named properties can be defined at provisioning time, but this list must not be static. Users must be able to add, delete, change new entries and create their own set of custom named properties as required.
Given that objects representation are similar to SQL database records, an architecture like the sqlite one makes sense for our purpose:
Component Brief description INTERFACE convenient top-level functions (Public API) accessed through EMSMDB providers PROCESSING set of atomic operations (open, read, write, close, mkdir, rmdir etc.) BACKENDS The code which does things in the specified storage system (mysql, fsocpf, imap etc.)
The interface layer doesn't have any knowledge about mapistore internals, how or where objects are stored. The interface uses MAPI data structure, supplies PR_FID and PR_MID values and assumes the interface layer will return a pack of data it can directly use without further or significant modifications. The interface layer functions should also have (as a parameter) a pointer to a mapistore context with private/opaque set of information (void *) about the object.
The processing layer is responsible for:
mapping OpenChange objects identifiers (PR_FID, PR_MID) to unique backends object identifiers (on purpose, depending on the kind of backend).
format input/output data: glue between INTERFACE and BACKENDS
relay input requests to the correct backend through atomic operations
maintain mapistore's integrity
The backends layer has a list of modules identified at mapistore initialization and available across user sessions, which means unique initialization at server start-up. Each module is a backend (fsocpf, sqlite, imap, etc.) and similarly to many other openchange components is loaded as a DSO object (dynamic shared object)
MAPIStore and the openchange 'dispatcher' database (openchange.ldb) are completely unrelated. MAPIStore is a standalone API and developers can use it independently from OpenChange server.
However, the mapistore API has initially been designed to be used by OpenChange server, and OpenChange server is using a tiny indexing database which describes user mailboxes top level containers. In openchange.ldb the mapistore_uri attribute is attached to top level containers and its value points to a valid mapistore URI (namespace + path). Note that a single user can have several different types of mapistore databases in use (one for each of the top level containers).
The is the only relationship between the database and the store: The database points to store locations.
MAPIStore needs to maintain a hash database linking unique OpenChange identifiers to unique backend identifiers. This hash table can be stored within a TDB database.
MAPIStore is responsible for managing IDs mapping between OpenChange objects and backend specific objects. It maintains a list of free identifiers which it reallocates on demand whenever a backend needs (mapistore_register_id()) one or where it wants to release one (mapistore_unregister_id()).
MAPIStore relies on the talloc library for memory allocation.
If there was a 'hello mapistore' program, it would only require to make 2 calls:
mapistore_init:
The initialization routine initializes the mapistore general context used along all mapistore calls, the mapping context databases (described below) and finally load all the backends available (DSO). When this operation is successful, developers are ready to make mapistore calls.
mapistore_release:
The release operation uninitializes the mapistore general context, closes connections to database and frees the remaining allocated memory.
mapistore_sample1.c
#include <mapistore/mapistore.h>
int main(int ac, const char *av[])
{
TALLOC_CTX *mem_ctx;
struct mapistore_context *mstore_ctx;
int retval;
/* Step 1. Create the talloc memory context */
mem_ctx = talloc_named(NULL, 0, "mapistore_sample1");
/* Step 2. Initialize mapistore system */
mstore_ctx = mapistore_init(mem_ctx, NULL);
if (!mstore_ctx) {
exit (1);
}
/* Step 3. Uninitialize mapistore system */
retval = mapistore_release(mstore_ctx);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
return 0;
}
$ export PKG_CONFIG_PATH=/usr/local/samba/lib/pkgconfig $ gcc mapistore_sample1.c -o mapistore_sample1 `pkg-config --cflags --libs libmapistore` $ ./mapistore_sample1 $
MAPIStore registers and loads its backends upon initialization. It means they are only instantiated/initialized one time during the whole server lifetime and the same code is used for all users and all mapistore folders.
These backend contexts (or connection contexts) are identified by a context id, which is an unsigned 32 bit integer which references the context during its lifetime. If OpenChange is used in a very large environment with many top folders (which implies the same number of mapistore contexts), or if OpenChange server has an incredibly long uptime, it would be possible to run out of available context identifiers.
In order to prevent this situation from happening, mapistore implements context databases where it stores available/free/used context identifiers:
mapistore_id_mapping_used.tdb: TDB database with used IDs
mapistore_id_mapping_free.tdb: TDB database with available pool of IDs
MAPIStore provides a convenient set of functions to manage backend contexts:
mapistore_set_mapping_path: Defines the path where context databases are stored. Call to this function is optional and default path would be used instead. However if a call to this function has to be made, it must be done before any call to mapistore (even mapistore_init).
mapistore_add_context: Add a new connection context to mapistore
mapistore_del_context: Delete a connection context from mapistore
mapistore_sample2.c:
#include <mapistore/mapistore.h>
int main(int ac, const char *av[])
{
TALLOC_CTX *mem_ctx;
struct mapistore_context *mstore_ctx;
int retval;
uint32_t context_id = 0;
uint32_t context_id2 = 0;
/* Step 1. Create the talloc memory context */
mem_ctx = talloc_named(NULL, 0, "mapistore_sample1");
/* Step 2. Set the mapping path to /tmp */
retval = mapistore_set_mapping_path("/tmp");
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
/* Step 3. Initialize mapistore system */
mstore_ctx = mapistore_init(mem_ctx, NULL);
if (!mstore_ctx) {
exit (1);
}
/* Step 4. Add connection contexts */
retval = mapistore_add_context(mstore_ctx, "fsocpf:///tmp/Inbox", &context_id);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
retval = mapistore_add_context(mstore_ctx, "fsocpf:///tmp/Sent Items", &context_id2);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
/* Step 5. Release connection contexts */
retval = mapistore_del_context(mstore_ctx, context_id);
retval = mapistore_del_context(mstore_ctx, context_id2);
/* Step 6. Uninitialize mapistore system */
retval = mapistore_release(mstore_ctx);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
return 0;
}
$ ./mapistore_sample2 sqlite3 backend initialized fsocpf backend initialized namespace is fsocpf:// and backend_uri is '/tmp/Inbox' [fsocpf_create_context:49] namespace is fsocpf:// and backend_uri is '/tmp/Sent Items' [fsocpf_create_context:49] $
FSOCPF stands for FileSystem and OpenChange Property Files. It is a backend designed to help developers testing OpenChange server code easily. The main idea is to have a backend we can manipulate, analyze and modify from the Linux console without having to develop a specific tool. This backend uses the UNIX filesystem for folder semantics and the OCPF file format as a way to store MAPI objects easily.
The namespace for this backend is:
fsocpf://
The mapistore_uri attribute for the folder definition in openchange.ldb must be a valid path where the last part of the URI is the FSOCPF container folder to create on the filesystem.
[+] Private user storage space
|
+-[+] Top-MAPIStore folder (Inbox)
|
+-[+] 0xf1000001 (mapistore folder1)
| |
| +-[+] .properties (OCPF)
| |
| +-[+] 0xe10000001.ocpf (message - OCPF)
| |
| +-[+] 0xe10000001 (attachment folder)
| |
| +-[+] 1.ocpf (PR_ATTACH_NUM)
| |
| +-[+] 1.data (attachment / stream data)
|
+-[+] 0xf2000001 (mapistore folder2)
|
+-[+] .properties (OCPF)
The figure above exposes the storage architecture of the FSOCPF backend using a real-world example. In this use case, we have decided to associate the FSOCPF backend to the Inbox folder. It means that any folder created under Inbox or any message stored within Inbox at any level of depth is stored and retrieved from the path defined in openchange.ldb.
In openchange.ldb, the mapistore_uri attribute of the Inbox record points to:
fsocpf://Private user storage space/Inbox
where Private user storage space can for example be
/usr/local/samba/private/mapistore/$username
Under Inbox, we have created 2 folders:
0xf1000001 folder1
0xf2000001 folder2
These folders are identified/named using their FID. Since they are classical filesystem folders, we can't associate attributes to them such as a folder comment, or the container class. All these attributes with the displayable name are stored into the .properties file within the folder.
Inside 0xf1000001 folder, we have 1 message named 0xe10000001.ocpf stored in the OCPF file format (property/value pair). Any properties associated to the message (subject, recipient, body) are stored within this file.
This message also has attachments. Attachments are stored within a directory at the same level named 0xe10000001 (message name without OCPF extension). Within this directory, we find the attachments named using the PR_ATTACH_NUM property value and the OCPF file extension. The content of the attachment is stored in $PR_ATTACH_NUM.data - in this case 1.data.
OpenChange Property File format (OCPF) documentation