libsent/src/util/mystrtok.c File Reference

Extract tokens from strings, with quotation handling. More...

#include <sent/stddefs.h>

Go to the source code of this file.

Defines
#define	ISTOKEN(A)   strchr(delim, A)
	Abbreviation macro to find if A is included in string delim.
Functions
char *	mystrtok_quotation (char *str, char *delim, int left_paren, int right_paren, int mode)
	Generic function to extract tokens from strings, with quotation handling.
char *	mystrtok_quote (char *str, char *delim)
	Extract tokens considering quotation by double quotation mark.
char *	mystrtok (char *str, char *delim)
	Extract tokens, not considering quotation, just as the same as strtok.
char *	mystrtok_movetonext (char *str, char *delim)
	Just move to the beginning of the next token, without modifying the str.

---

Detailed Description

Extract tokens from strings, with quotation handling.

When extracting tokens from strings, the part enclosed by the specified braces are forced to be treated as a single token.

Author:

Akinobu LEE

Date:

Thu Feb 17 16:31:39 2005

Revision

1.1.1.1

Definition in file mystrtok.c.

---

Function Documentation

char* mystrtok_quotation	(	char *	str,
		char *	delim,
		int	left_paren,
		int	right_paren,
		int	mode
	)

Generic function to extract tokens from strings, with quotation handling.

The usage is as the same as strtok.

Parameters:

	str	[i/o] source string, or NULL when this is a continuous call from previous call. Will be truncated in this function.
	delim	[in] string to specify the delimiters.
	left_paren	[in] left brace
	right_paren	[in] right brace
	mode	[in] if 1, just move to the beginning of next token

Returns:

pointer to the next extracted token.

Definition at line 51 of file mystrtok.c.

Referenced by mystrtok(), mystrtok_movetonext(), mystrtok_quote(), voca_load_htkdict_line(), and voca_load_wordlist_line().

char* mystrtok_quote	(	char *	str,
		char *	delim
	)

Extract tokens considering quotation by double quotation mark.

Parameters:

	str	[i/o] source string, will be truncated.
	delim	[in] string of all token delimiters

Returns:

pointer to the next extracted token, or NULL when no token found.

Definition at line 107 of file mystrtok.c.

Referenced by read_token(), voca_load_htkdict_line(), and voca_load_wordlist_line().

char* mystrtok	(	char *	str,
		char *	delim
	)

Extract tokens, not considering quotation, just as the same as strtok.

Parameters:

	str	[i/o] source string, will be truncated.
	delim	[in] string of all token delimiters

Returns:

pointer to the next extracted token, or NULL when no token found.

Definition at line 121 of file mystrtok.c.

Referenced by voca_load_htkdict_line(), and voca_load_wordlist_line().

char* mystrtok_movetonext	(	char *	str,
		char *	delim
	)

Just move to the beginning of the next token, without modifying the str.

Parameters:

	str	[i/o] source string, will be truncated.
	delim	[in] string of all token delimiters

Returns:

pointer to the next extracted token, or NULL when no token found.

Definition at line 135 of file mystrtok.c.

Referenced by voca_load_htkdict_line(), and voca_load_wordlist_line().

---