Three-staged, multithreaded update pipeline. More...

#include <UpdatePipeline.h>

Collaboration diagram for Hypertable::UpdatePipeline:

Public Member Functions
	UpdatePipeline (ContextPtr &context, QueryCachePtr &query_cache, TimerHandlerPtr &timer_handler, CommitLogPtr &log, Filesystem::Flags flags)
	Constructor. More...

void	add (UpdateContext *uc)
	Adds updates to pipeline Adds `uc` to m_qualify_queue and signals m_qualify_queue_cond. More...

void	shutdown ()
	Shuts down the pipeline Sets m_shutdown to true, signals the three pipeline condition variables, and performs a join on each pipeline thread. More...

Private Member Functions
void	qualify_and_transform ()
	Thread function for stage 1 of update pipeline. More...

void	commit ()
	Thread function for stage 2 of update pipeline. More...

void	add_and_respond ()
	Thread function for stage 3 of update pipeline. More...

void	transform_key (ByteString &bskey, DynamicBuffer dest_bufp, int64_t revision, int64_t revisionp, bool timeorder_desc)

Private Attributes
std::shared_ptr< Context >	m_context
	Range server context More...

QueryCachePtr	m_query_cache
	Pointer to query cache. More...

TimerHandlerPtr	m_timer_handler
	Pointer to timer handler. More...

CommitLogPtr	m_log {}
	Pointer to commit log. More...

std::mutex	m_qualify_queue_mutex
	Mutex protecting stage 1 input queue More...

std::condition_variable	m_qualify_queue_cond
	Condition variable signaling addition to stage 1 input queue. More...

std::list< UpdateContext * >	m_qualify_queue
	Stage 1 input queue. More...

std::mutex	m_commit_queue_mutex
	Mutex protecting stage 2 input queue More...

std::condition_variable	m_commit_queue_cond
	Condition variable signaling addition to stage 2 input queue. More...

int32_t	m_commit_queue_count {}
	Count of objects in stage 2 input queue. More...

std::list< UpdateContext * >	m_commit_queue
	Stage 2 input queue. More...

std::mutex	m_response_queue_mutex
	Mutex protecting stage 3 input queue More...

std::condition_variable	m_response_queue_cond
	Condition variable signaling addition to stage 3 input queue. More...

std::list< UpdateContext * >	m_response_queue
	Stage 3 input queue. More...

std::vector< std::thread >	m_threads
	Update pipeline threads. More...

int64_t	m_last_revision {TIMESTAMP_MIN}
	Last (largest) assigned revision number. More...

uint64_t	m_update_coalesce_limit {}
	Commit log coalesce limit. More...

int32_t	m_maintenance_pause_interval {}
	Millisecond pause time at the end of the pipeline (TESTING) More...

uint32_t	m_update_delay {}
	Update delay at start of pipeline (TESTING) More...

int32_t	m_max_clock_skew {}
	Maximum allowable clock skew. More...

Filesystem::Flags	m_flags {}
	Commit log flush flag (NONE, FLUSH, or SYNC) More...

bool	m_shutdown {}
	Flag indicating if pipeline is being shut down. More...

Detailed Description

Three-staged, multithreaded update pipeline.

Definition at line 52 of file UpdatePipeline.h.

Constructor & Destructor Documentation

UpdatePipeline::UpdatePipeline	(	ContextPtr &	context,
		QueryCachePtr &	query_cache,
		TimerHandlerPtr &	timer_handler,
		CommitLogPtr &	log,
		Filesystem::Flags	flags
	)

Constructor.

Initializes the pipeline as follows:

Sets m_update_coalesce_limit to the value of the Hypertable.RangeServer.UpdateCoalesceLimit property.
Sets m_maintenance_pause_interval to the value of the Hypertable.RangeServer.Testing.MaintenanceNeeded.PauseInterval property.
Sets m_update_delay to the value of the Hypertable.RangeServer.UpdateDelay property.
Sets m_max_clock_skew to the value of the Hypertable.RangeServer.ClockSkew.Max property.
Creates and starts the three pipeline threads using qualify_and_transform(), commit(), and add_and_respond() as the thread functions, respectively.
Parameters

context Range server context

query_cache Query cache

timer_handler Timer handler

Definition at line 52 of file UpdatePipeline.cc.

Member Function Documentation

void UpdatePipeline::add ( UpdateContext * uc )

Adds updates to pipeline Adds uc to m_qualify_queue and signals m_qualify_queue_cond.

Parameters

uc	Update context

Definition at line 67 of file UpdatePipeline.cc.

void UpdatePipeline::add_and_respond ( )

private

Thread function for stage 3 of update pipeline.

For each UpdateContext object on the input queue m_response_queue, this function does the following:

Adds the key/value pairs that were commited in the previous state to their appropriate ranges
Sends back a response to the originating requests

Insert updates into Ranges

wait for these ranges to complete maintenance

Send back response

Definition at line 651 of file UpdatePipeline.cc.

void UpdatePipeline::commit ( )

private

Thread function for stage 2 of update pipeline.

For each UpdateContext object on the input queue m_commit_queue, this function does the following:

Writes the key/value pairs that were buffered in the previous stage to the appropriate commit log (or transfer log) without calling sync().
Once either m_update_coalesce_limit amount of updates has been collected or when m_qualify_queue becomes empty, sync() is called on the commit (or transfer) log.
Adds the UpdateContext objects to m_response_queue and signals m_response_queue_cond.

Definition at line 516 of file UpdatePipeline.cc.

void UpdatePipeline::qualify_and_transform ( )

private

Thread function for stage 1 of update pipeline.

For each UpdateContext object on the input queue m_qualify_queue, this function does the following:

Extracts the updates that are destined for this range server
Creates a SendBackRec for the key/value pairs not destined for this range server.
Transforms each key with a call to transform_key().
Buffers the key/value pairs for downstream processing.
Adds the UpdateContext objects to m_commit_queue and signals m_commit_queue_cond.

Fetch range transfer information

Definition at line 85 of file UpdatePipeline.cc.

void UpdatePipeline::shutdown ( )

Shuts down the pipeline Sets m_shutdown to true, signals the three pipeline condition variables, and performs a join on each pipeline thread.

Definition at line 75 of file UpdatePipeline.cc.

void UpdatePipeline::transform_key	(	ByteString &	bskey,
		DynamicBuffer *	dest_bufp,
		int64_t	revision,
		int64_t *	revisionp,
		bool	timeorder_desc
	)

private

Definition at line 827 of file UpdatePipeline.cc.

Member Data Documentation

std::list<UpdateContext *> Hypertable::UpdatePipeline::m_commit_queue

private

Stage 2 input queue.

Definition at line 156 of file UpdatePipeline.h.

std::condition_variable Hypertable::UpdatePipeline::m_commit_queue_cond

private

Condition variable signaling addition to stage 2 input queue.

Definition at line 150 of file UpdatePipeline.h.

int32_t Hypertable::UpdatePipeline::m_commit_queue_count {}

private

Count of objects in stage 2 input queue.

Definition at line 153 of file UpdatePipeline.h.

std::mutex Hypertable::UpdatePipeline::m_commit_queue_mutex

private

Mutex protecting stage 2 input queue

Definition at line 147 of file UpdatePipeline.h.

std::shared_ptr<Context> Hypertable::UpdatePipeline::m_context

private

Range server context

Definition at line 126 of file UpdatePipeline.h.

Filesystem::Flags Hypertable::UpdatePipeline::m_flags {}

private

Commit log flush flag (NONE, FLUSH, or SYNC)

Definition at line 186 of file UpdatePipeline.h.

int64_t Hypertable::UpdatePipeline::m_last_revision {TIMESTAMP_MIN}

private

Last (largest) assigned revision number.

Definition at line 171 of file UpdatePipeline.h.

CommitLogPtr Hypertable::UpdatePipeline::m_log {}

private

Pointer to commit log.

Definition at line 135 of file UpdatePipeline.h.

int32_t Hypertable::UpdatePipeline::m_maintenance_pause_interval {}

private

Millisecond pause time at the end of the pipeline (TESTING)

Definition at line 177 of file UpdatePipeline.h.

int32_t Hypertable::UpdatePipeline::m_max_clock_skew {}

private

Maximum allowable clock skew.

Definition at line 183 of file UpdatePipeline.h.

std::list<UpdateContext *> Hypertable::UpdatePipeline::m_qualify_queue

private

Stage 1 input queue.

Definition at line 144 of file UpdatePipeline.h.

std::condition_variable Hypertable::UpdatePipeline::m_qualify_queue_cond

private

Condition variable signaling addition to stage 1 input queue.

Definition at line 141 of file UpdatePipeline.h.

std::mutex Hypertable::UpdatePipeline::m_qualify_queue_mutex

private

Mutex protecting stage 1 input queue

Definition at line 138 of file UpdatePipeline.h.

QueryCachePtr Hypertable::UpdatePipeline::m_query_cache

private

Pointer to query cache.

Definition at line 129 of file UpdatePipeline.h.

std::list<UpdateContext *> Hypertable::UpdatePipeline::m_response_queue

private

Stage 3 input queue.

Definition at line 165 of file UpdatePipeline.h.

std::condition_variable Hypertable::UpdatePipeline::m_response_queue_cond

private

Condition variable signaling addition to stage 3 input queue.

Definition at line 162 of file UpdatePipeline.h.

std::mutex Hypertable::UpdatePipeline::m_response_queue_mutex

private

Mutex protecting stage 3 input queue

Definition at line 159 of file UpdatePipeline.h.

bool Hypertable::UpdatePipeline::m_shutdown {}

private

Flag indicating if pipeline is being shut down.

Definition at line 189 of file UpdatePipeline.h.

std::vector<std::thread> Hypertable::UpdatePipeline::m_threads

private

Update pipeline threads.

Definition at line 168 of file UpdatePipeline.h.

TimerHandlerPtr Hypertable::UpdatePipeline::m_timer_handler

private

Pointer to timer handler.

Definition at line 132 of file UpdatePipeline.h.

uint64_t Hypertable::UpdatePipeline::m_update_coalesce_limit {}

private

Commit log coalesce limit.

Definition at line 174 of file UpdatePipeline.h.

uint32_t Hypertable::UpdatePipeline::m_update_delay {}

private

Update delay at start of pipeline (TESTING)

Definition at line 180 of file UpdatePipeline.h.

The documentation for this class was generated from the following files:

/home/doug/src/hypertable/src/cc/Hypertable/RangeServer/UpdatePipeline.h
/home/doug/src/hypertable/src/cc/Hypertable/RangeServer/UpdatePipeline.cc

context	Range server context
query_cache	Query cache
timer_handler	Timer handler

Public Member Functions

Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation