#include <sent/stddefs.h>
#include <sent/ngram2.h>
Include dependency graph for ngram_read_bin.c:
Go to the source code of this file.
Functions | |
static void | rdn (FILE *fp, void *buf, size_t unitbyte, int unitnum) |
static void | check_header (FILE *fp) |
boolean | ngram_read_bin (FILE *fp, NGRAM_INFO *ndata) |
Variables | |
static int | file_version |
N-gram format version of the file. | |
static boolean | need_swap |
TRUE if need byte swap. |
From 3.5, internal format of binary N-gram has changed for using machine-dependent natural byte order (previously fixed to big endian), 24bit index and 2-gram backoff compression. So, binary N-gram generated by mkbingram of 3.5 and later will not work on 3.4.2 and earlier versions.
There is full upward- and cross-machine compatibility in 3.5. Old binary N-gram files still can be read directly, in which case the conversion to 24bit index will performed just after model has been read. Byte order will also considered by header information, so binary N-gram still can be used among different machines.
Definition in file ngram_read_bin.c.
static void rdn | ( | FILE * | fp, | |
void * | buf, | |||
size_t | unitbyte, | |||
int | unitnum | |||
) | [static] |
Binary read function with byte swap
fp | [in] file pointer | |
buf | [out] data buffer | |
unitbyte | [in] unit size in bytes | |
unitnum | [in] number of unit to read. |
Definition at line 79 of file ngram_read_bin.c.
static void check_header | ( | FILE * | fp | ) | [static] |
Check header to see whether the version matches.
fp | [in] file pointer |
Definition at line 134 of file ngram_read_bin.c.
Referenced by ngram_read_bin().
boolean ngram_read_bin | ( | FILE * | fp, | |
NGRAM_INFO * | ndata | |||
) |
Read a N-gram binary file and store to data.
fp | [in] file pointer | |
ndata | [out] N-gram data to store the read data |
Definition at line 230 of file ngram_read_bin.c.
Referenced by init_ngram_bin().