Writes a MetaLog. More...

#include <MetaLogWriter.h>

Collaboration diagram for Hypertable::MetaLog::Writer:

Classes
class	WriteScheduler
	Periodically flushes deferred writes to disk. More...

Public Member Functions
	Writer (FilesystemPtr &fs, DefinitionPtr &definition, const std::string &path, std::vector< EntityPtr > &initial_entities)
	Constructor. More...

	~Writer ()
	Destructor. More...

void	close ()
	Closes open file descriptors. More...

void	record_state (EntityPtr entity)
	Persists an Entity to the log. More...

void	record_state (std::vector< EntityPtr > &entities)
	Persists a vector of entities to the log. More...

void	record_removal (EntityPtr entity)
	Records the removal of an entity. More...

void	record_removal (std::vector< EntityPtr > &entities)
	Records the removal of a vector of entities. More...

void	signal_write_ready ()

Static Public Attributes
static bool	skip_recover_entry = false
	Global flag to force writer to skip writing EntityRecover (testing) More...

Private Types
typedef std::shared_ptr < WriteScheduler >	WriteSchedulerPtr

typedef std::pair< size_t, std::shared_ptr< uint8_t > >	SerializedEntityT

Private Member Functions
void	write_header ()
	Writes MetaLog file header. More...

void	purge_old_log_files ()
	Purges old MetaLog files. More...

void	roll ()

void	service_write_queue ()

Private Attributes
std::mutex	m_mutex
	Mutex for serializing access to members More...

condition_variable	m_cond
	Condition variable to signal completion of deferred writes. More...

FilesystemPtr	m_fs
	Smart pointer to Filesystem object. More...

DefinitionPtr	m_definition
	Smart pointer to MetaLog Definition. More...

std::string	m_path
	Path name of MetaLog directory. More...

std::string	m_filename
	Full pathname of MetaLog file open for writing. More...

int	m_fd
	File descriptor of MetaLog file in FS. More...

std::string	m_backup_path
	Pathname of local log backup directory. More...

std::string	m_backup_filename
	Pathname of local log backup file. More...

int	m_backup_fd
	File descriptor of backup MetaLog file in local filesystem. More...

int32_t	m_offset {}
	Current write offset of MetaLog file. More...

std::deque< int32_t >	m_file_ids
	Deque of existing file name IDs. More...

int64_t	m_max_file_size {}
	Maximum file size. More...

size_t	m_history_size {}
	Number of old MetaLog files to retain for historical purposes. More...

int32_t	m_replication {}
	Replication factor. More...

Filesystem::Flags	m_flush_method {}
	Log flush method (FLUSH or SYNC) More...

std::map< int64_t, SerializedEntityT >	m_entity_map
	Map of current serialized entity data. More...

bool	m_write_ready {}
	Flag indicating that. More...

std::vector< StaticBufferPtr >	m_write_queue
	Vector of pending writes. More...

WriteSchedulerPtr	m_write_scheduler
	Write scheduler. More...

Detailed Description

Writes a MetaLog.

This class is used to persist application entities to a MetaLog. It is constructed with an initial set of entities to persist (obtained from a prior read of the log) and is kept open during the duration of the server process and is used to persist state changes by appending updated Entity objects. It is typically constructed as follows:

MetaLog::Reader reader = new MetaLog::Reader(log_fs, definition, log_dir);
vector<MetaLog::EntityPtr> entities;
reader->get_entities(entities);
MetaLog::Writer writer = new MetaLog::Writer(log_fs, definition, log_dir, entities);

Definition at line 66 of file MetaLogWriter.h.

Member Typedef Documentation

typedef std::pair<size_t, std::shared_ptr<uint8_t> > Hypertable::MetaLog::Writer::SerializedEntityT

private

Definition at line 263 of file MetaLogWriter.h.

typedef std::shared_ptr<WriteScheduler> Hypertable::MetaLog::Writer::WriteSchedulerPtr

private

Definition at line 187 of file MetaLogWriter.h.

Constructor & Destructor Documentation

Writer::Writer	(	FilesystemPtr &	fs,
		DefinitionPtr &	definition,
		const std::string &	path,
		std::vector< EntityPtr > &	initial_entities
	)

Constructor.

Initializes the writer by setting m_path to path and creating it if it doesn't exist. It then initializes m_backup_path (creating it in the local filesystem if it does not exist) as follows:

$data_dir     = Config::properties->get_str("Hypertable.DataDirectory");
$name         = m_definition->name();
$backup_label = m_definition->backup_label()
m_backup_path = $data_dir/run/log_backup/$name/$backup_label

It then scans the log directory with a call to scan_log_directory() to obtain the current set of numeric file names in the log directory (path) and the next unused numeric file name which is stored in local variable, next_id. Old log files are purged with a call to purge_old_log_files() with a "keep count" of 30. It then sets m_filename equal to path/next_id and opens it for writing, setting the m_fd to the opened file descriptor. The replication factor is set to the Hypertable.Metadata.Replication property. It then sets m_backup_filename to m_backup_path + \/next_id and opens it for writing (in the local filesystem), setting m_backup_fd to the opened file descriptor. It then writes the file header with a call to write_header() and persists initial_entities. Finally, it writes a RecoverEntity.

Parameters

fs	Smart pointer to Filesystem object
definition	Smart pointer to Definition object
path	Path to MetaLog directory
initial_entities	Initial set of entities to write into log

Definition at line 64 of file MetaLogWriter.cc.

Writer::~Writer ( )

Destructor.

Calls close().

Definition at line 122 of file MetaLogWriter.cc.

Member Function Documentation

void Writer::close ( )

Closes open file descriptors.

This method closes both m_fd and m_backup_fd and sets them to -1.

Definition at line 127 of file MetaLogWriter.cc.

void Writer::purge_old_log_files ( )

private

Purges old MetaLog files.

This method removes the MetaLog files with the numerically smallest names until the number of remaining files is equal to m_history_size. The files are removed from the FS as well as the backup directory. The m_file_ids member is assumed to be populated on entry with the file name IDs in the log directory and is adjusted to only include the file name IDs that remain after purging.

Definition at line 144 of file MetaLogWriter.cc.

void Writer::record_removal ( EntityPtr entity )

Records the removal of an entity.

This method records the removal of entity by setting the EntityHeader::FLAG_REMOVE bit in the flags field of the header, setting the length and checksum header fields to 0, and then writing the entity header. First the entity header is appended to the local backup file and then to the log file in the FS. m_offset is incremented by the length of the serialized entity.

Parameters

entity Pointer to entity to remove

Definition at line 412 of file MetaLogWriter.cc.

void Writer::record_removal ( std::vector< EntityPtr > & entities )

Records the removal of a vector of entities.

This method is logically equivalent to calling single_entity_record_removal for each entity in entities.

Parameters

entities Reference to vector of entities to remove

Definition at line 441 of file MetaLogWriter.cc.

void Writer::record_state ( EntityPtr entity )

Persists an Entity to the log.

If the Entity::marked_for_removal() method of entity returns true, just the entity header is persisted, otherwise the entity header and state are persisted with a call to the Entity::encode_entry() method. First the entity is appended to the local backup file and then to the log file in the FS. m_offset is incremented by the length of the serialized entity.

Parameters

entity Pointer to entity to persist

Definition at line 315 of file MetaLogWriter.cc.

void Writer::record_state ( std::vector< EntityPtr > & entities )

Persists a vector of entities to the log.

This method is logically equivalent to calling single_entity_record_state for each entity in entities.

Parameters

entities Reference to vector of entities to persist

Definition at line 356 of file MetaLogWriter.cc.

void Writer::roll ( )

private

Definition at line 162 of file MetaLogWriter.cc.

void Writer::service_write_queue ( )

private

Definition at line 215 of file MetaLogWriter.cc.

void Writer::signal_write_ready ( )

Definition at line 476 of file MetaLogWriter.cc.

void Writer::write_header ( )

private

Writes MetaLog file header.

This method writes a MetaLog file header to the open MetaLog file by initializing a MetaLog::Header object with the name and version obtained by calling the methods Definition::name() and Definition::version() of m_definition. First the file header is appended to the local backup file and then to the log file in the FS. m_offset is incremented by the length of the serialized file header (Header::LENGTH).

Definition at line 287 of file MetaLogWriter.cc.

Member Data Documentation

int Hypertable::MetaLog::Writer::m_backup_fd

private

File descriptor of backup MetaLog file in local filesystem.

Definition at line 242 of file MetaLogWriter.h.

std::string Hypertable::MetaLog::Writer::m_backup_filename

private

Pathname of local log backup file.

Definition at line 239 of file MetaLogWriter.h.

std::string Hypertable::MetaLog::Writer::m_backup_path

private

Pathname of local log backup directory.

Definition at line 236 of file MetaLogWriter.h.

condition_variable Hypertable::MetaLog::Writer::m_cond

private

Condition variable to signal completion of deferred writes.

Definition at line 218 of file MetaLogWriter.h.

DefinitionPtr Hypertable::MetaLog::Writer::m_definition

private

Smart pointer to MetaLog Definition.

Definition at line 224 of file MetaLogWriter.h.

std::map<int64_t, SerializedEntityT> Hypertable::MetaLog::Writer::m_entity_map

private

Map of current serialized entity data.

Definition at line 266 of file MetaLogWriter.h.

int Hypertable::MetaLog::Writer::m_fd

private

File descriptor of MetaLog file in FS.

Definition at line 233 of file MetaLogWriter.h.

std::deque<int32_t> Hypertable::MetaLog::Writer::m_file_ids

private

Deque of existing file name IDs.

Definition at line 248 of file MetaLogWriter.h.

std::string Hypertable::MetaLog::Writer::m_filename

private

Full pathname of MetaLog file open for writing.

Definition at line 230 of file MetaLogWriter.h.

Filesystem::Flags Hypertable::MetaLog::Writer::m_flush_method {}

private

Log flush method (FLUSH or SYNC)

Definition at line 260 of file MetaLogWriter.h.

FilesystemPtr Hypertable::MetaLog::Writer::m_fs

private

Smart pointer to Filesystem object.

Definition at line 221 of file MetaLogWriter.h.

size_t Hypertable::MetaLog::Writer::m_history_size {}

private

Number of old MetaLog files to retain for historical purposes.

Definition at line 254 of file MetaLogWriter.h.

int64_t Hypertable::MetaLog::Writer::m_max_file_size {}

private

Maximum file size.

Definition at line 251 of file MetaLogWriter.h.

std::mutex Hypertable::MetaLog::Writer::m_mutex

private

Mutex for serializing access to members

Definition at line 215 of file MetaLogWriter.h.

int32_t Hypertable::MetaLog::Writer::m_offset {}

private

Current write offset of MetaLog file.

Definition at line 245 of file MetaLogWriter.h.

std::string Hypertable::MetaLog::Writer::m_path

private

Path name of MetaLog directory.

Definition at line 227 of file MetaLogWriter.h.

int32_t Hypertable::MetaLog::Writer::m_replication {}

private

Replication factor.

Definition at line 257 of file MetaLogWriter.h.

std::vector<StaticBufferPtr> Hypertable::MetaLog::Writer::m_write_queue

private

Vector of pending writes.

Definition at line 272 of file MetaLogWriter.h.

bool Hypertable::MetaLog::Writer::m_write_ready {}

private

Flag indicating that.

Definition at line 269 of file MetaLogWriter.h.

WriteSchedulerPtr Hypertable::MetaLog::Writer::m_write_scheduler

private

Write scheduler.

Definition at line 275 of file MetaLogWriter.h.

bool Writer::skip_recover_entry = false

static

Global flag to force writer to skip writing EntityRecover (testing)

Definition at line 155 of file MetaLogWriter.h.

The documentation for this class was generated from the following files:

/home/doug/src/hypertable/src/cc/Hypertable/Lib/MetaLogWriter.h
/home/doug/src/hypertable/src/cc/Hypertable/Lib/MetaLogWriter.cc

Classes

Public Member Functions

Static Public Attributes

Private Types

Private Member Functions

Private Attributes

Detailed Description

Member Typedef Documentation

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation