libsent/src/util/mystrtok.c File Reference

Extract tokens from strings, with quotation handling. More...

#include <sent/stddefs.h>

Include dependency graph for mystrtok.c:

Go to the source code of this file.

Defines
#define	ISTOKEN(A)   strchr(delim, A)
	Abbreviation macro to find if A is included in string delim.
Functions
char *	mystrtok_quotation (char *str, char *delim, int left_paren, int right_paren, int mode)
char *	mystrtok_quote (char *str, char *delim)
char *	mystrtok (char *str, char *delim)
char *	mystrtok_movetonext (char *str, char *delim)

---

Detailed Description

Extract tokens from strings, with quotation handling.

Author:

Akinobu LEE

Date:

Thu Feb 17 16:31:39 2005

When extracting tokens from strings, the part enclosed by the specified braces are forced to be treated as a single token.

Revision

1.3

Definition in file mystrtok.c.

---

Function Documentation

char* mystrtok_quotation	(	char *	str,
		char *	delim,
		int	left_paren,
		int	right_paren,
		int	mode
	)

Generic function to extract tokens from strings, with quotation handling. The usage is as the same as strtok.

Parameters:

	str	[i/o] source string, or NULL when this is a continuous call from previous call. Will be truncated in this function.
	delim	[in] string to specify the delimiters.
	left_paren	[in] left brace
	right_paren	[in] right brace
	mode	[in] if 1, just move to the beginning of next token

Returns:

pointer to the next extracted token.

Definition at line 50 of file mystrtok.c.

Referenced by mystrtok(), mystrtok_movetonext(), mystrtok_quote(), and voca_load_htkdict_line().

char* mystrtok_quote	(	char *	str,
		char *	delim
	)

Extract tokens considering quotation by double quotation mark.

Parameters:

	str	[i/o] source string, will be truncated.
	delim	[in] string of all token delimiters

Returns:

pointer to the next extracted token, or NULL when no token found.

Definition at line 104 of file mystrtok.c.

Referenced by read_token().

char* mystrtok	(	char *	str,
		char *	delim
	)

Extract tokens, not considering quotation, just as the same as strtok.

Parameters:

	str	[i/o] source string, will be truncated.
	delim	[in] string of all token delimiters

Returns:

pointer to the next extracted token, or NULL when no token found.

Definition at line 118 of file mystrtok.c.

Referenced by voca_load_htkdict_line().

char* mystrtok_movetonext	(	char *	str,
		char *	delim
	)

Just move to the beginning of the next token, without modifying the str.

Parameters:

	str	[i/o] source string, will be truncated.
	delim	[in] string of all token delimiters

Returns:

pointer to the next extracted token, or NULL when no token found.

Definition at line 132 of file mystrtok.c.

Referenced by voca_load_htkdict_line().

---