Splits cluster definition file into tokens. More...

#include <Tokenizer.h>

Collaboration diagram for Hypertable::ClusterDefinitionFile::Tokenizer:

Public Member Functions
	Tokenizer (const string &fname)
	Constructor. More...

	Tokenizer (const string &fname, const string &content)
	Constructor. More...

string	dirname ()
	Returns the directory path containing the cluster definition file. More...

bool	next (Token &token)
	Returns the next token from the cluster definition file. More...

Private Member Functions
int	identify_line_type (const char base, const char end)
	Identifies token type of line starting at `base`. More...

bool	accumulate (const char *basep, const char end, int type, Token &token)
	Accumulates the next token. More...

Private Attributes
string	m_fname
	Pathname of cluster definition file. More...

string	m_content
	Content of cluster definition file. More...

const char *	m_next
	Pointer to beginning of next token to read. More...

size_t	m_line {}
	Line number of end of last token read. More...

Detailed Description

Splits cluster definition file into tokens.

Definition at line 43 of file Tokenizer.h.

Constructor & Destructor Documentation

Tokenizer::Tokenizer ( const string & fname )

Constructor.

Initializes m_fname with fname, reads the contents of m_fname into m_content, and then sets m_next pointing to the beginning of the content.

Parameters

fname Pathname of cluster definition file

Definition at line 54 of file Tokenizer.cc.

Tokenizer::Tokenizer	(	const string &	fname,
		const string &	content
	)

Constructor.

Initializes m_fname with fname, sets m_content to content, and then sets m_next pointing to the beginning of the content.

Parameters

fname	Pathname of cluster definition file
content	Content of cluster definition file

Definition at line 61 of file Tokenizer.cc.

Member Function Documentation

bool Tokenizer::accumulate	(	const char **	basep,
		const char *	end,
		int	type,
		Token &	token
	)

private

Accumulates the next token.

This function populates token with the next token to be returned. If token is already populated, it first check to see if the following token (identified by basep, end, and type) should be merged with the existing token and if so, it merges the token text with the existing token and advances *basep to end+1, otherwise it leaves *basep unmodified and returns. Tokens that get merged are Token::COMMENT followed by Token::TASK, Token::ROLE followed by Token::CODE, and any a adjacent code tokens (e.g. Token::CODE, Token::FUNCTION, Token::BLANKLINE, Token::CONTROLFLOW). If token is populated with the token text starting at *basep because it was merged with an existing token or the existing token was empty, then it sets m_next to end.

Parameters

basep	Address of pointer to next token
end	Pointer to end of next token (either newline or EOF)
type	Type of next token (see Token::Type)
token	Reference to token object to hold next token

Returns: true if token is ready to be returned

Definition at line 251 of file Tokenizer.cc.

string Tokenizer::dirname ( )

Returns the directory path containing the cluster definition file.

Returns the path of m_fname up to the last '/' character, or "." if m_fname does not contain any '/' character.

Returns: The directory path containing the cluster definition file.

Definition at line 66 of file Tokenizer.cc.

int Tokenizer::identify_line_type	(	const char *	base,
		const char *	end
	)

private

Identifies token type of line starting at base.

This function determines the token type (Token::Type) of the line starting at base and ending at end.

Parameters

base	Pointer to beginning of next line
end	Pointer to end of line or end of content

Returns: Token type of line pointed to by base

See also: Token::Type

Definition at line 190 of file Tokenizer.cc.

bool Tokenizer::next ( Token & token )

Returns the next token from the cluster definition file.

This reads the next token from the cluster definition file starting at m_next. It populates token with the token text, type identifier, line number of the start of the token, and creates a translator for it with a call to Token::create_translator(). The m_next and m_line are advanced to the beginning of the next token (or EOF) and the last line of the returned token, respectively.

Parameters

token Next token filled in by this function

Returns: true token was read, false on EOF.

Definition at line 73 of file Tokenizer.cc.

Member Data Documentation

string Hypertable::ClusterDefinitionFile::Tokenizer::m_content

private

Content of cluster definition file.

Definition at line 115 of file Tokenizer.h.

string Hypertable::ClusterDefinitionFile::Tokenizer::m_fname

private

Pathname of cluster definition file.

Definition at line 113 of file Tokenizer.h.

size_t Hypertable::ClusterDefinitionFile::Tokenizer::m_line {}

private

Line number of end of last token read.

Definition at line 119 of file Tokenizer.h.

const char* Hypertable::ClusterDefinitionFile::Tokenizer::m_next

private

Pointer to beginning of next token to read.

Definition at line 117 of file Tokenizer.h.

The documentation for this class was generated from the following files:

/home/doug/src/hypertable/src/cc/Common/ClusterDefinitionFile/Tokenizer.h
/home/doug/src/hypertable/src/cc/Common/ClusterDefinitionFile/Tokenizer.cc

Public Member Functions

Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation