     START-INFO-DIR-ENTRY
     * Libbow::                      Bag-Of-Words Library
     END-INFO-DIR-ENTRY

     [file version.texi not included]

This file documents the features and implementation of Libbow.

Copyright (C) 1996, 1997, 1998 Free Software Foundation, Inc.

Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies.

Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided also that the section entitled ``GNU Library General Public License'' is included exactly as in the original, and provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions, except that the section entitled ``GNU Library General Public License'' and this permission notice may be included in translations approved by the Free Software Foundation instead of in the original English.

This manual documents how to install and use libbow), (a library of C code for statistical text processing), version BOWVERSION.

Overview
********

     [file libbow-desc.texi not included]

Pronounciation guide: "libbow" rhymes with "lib-low", not "lib-cow".

Notes from Devika:

How to delimit documents. How to tag things---how to augment the lexers. Lead in gently, steps.  Big picture.... more and more interesting things Variety of examples. Guide to sea of command-line references.  Structure. When to consider using which switch. Sensible defaults.

Traversing Diretories to find Text Files
****************************************

Getting Words from Text Files
*****************************

Lexer buffers, Lexers

The Simple Lexer
================

The N-Gram Lexer
================

The Email/News Lexer
====================

The HTML Lexer
==============

Functions Useful for Writing Lexers
===================================

 -- Function: int bow_stem_porter (char *WORD)
     Apply the Porter stemming algorithm to modify WORD.  Return 0 on success.

 -- Function: int bow_isalpha (int CHARACTER)
     A function wrapper around POSIX's isalpha macro.

 -- Function: int bow_isgraph (int CHARACTER)
     A function wrapper around POSIX's isgraph macro.

 -- Function: int bow_stoplist_present (const char *WORD)
     Return non-zero if WORD is on the stoplist.

 -- Function: int bow_stoplist_add_from_file (const char *FILENAME)
     Add to the stoplist the white-space delineated words from FILENAME.  Return the number of words added.  If the file could not be opened, return -1.

Mapping between Words and Integers
**********************************

Generic Maps between Integers and Strings
=========================================

 -- bow_int4str:

 -- Function: bow_int4str * bow_int4str_new (int CAPACITY)
     Allocate, initialize and return a new int/string mapping structure.  The parameter CAPACITY is used as a hint about the number of words to expect; if you don't know or don't care about a CAPACITY value, pass 0, and a default value will be used.

 -- Function: const char * bow_int2str (bow_int4str *MAP, int INDEX)
     Given a integer INDEX, return its corresponding string.

 -- Function: int bow_str2int (bow_int4str *MAP, const char *STRING)
     Given the char-pointer STRING, return its integer index.  If this is the first time we're seeing STRING, add it to the mapping, assign it a new index, and return the new index.

 -- Function: int bow_str2int_no_add (bow_int4str *MAP, const char *STRING)
     Given the char-pointer STRING, return its integer index.  If STRING is not yet in the mapping, return -1.

 -- Function: void bow_int4str_write (bow_int4str *MAP, FILE *FP)
     Write the int-str mapping to file-pointer FP.

 -- Function: bow_int4str * bow_int4str_new_from_fp (FILE *FP)
     Return a new int-str mapping, created by reading file-pointer FP.

 -- Function: bow_int4str * bow_int4str_new_from_file (const char *FILENAME)
     Return a new int-string mapping, created by reading FILENAME.

 -- Function: void bow_int4str_free (bow_int4str *MAP)
     Free the memory held by the int-string mapping MAP.

The Global Dictionary
=====================

 -- Function: const char * bow_int2word (int WI)
     Given a "word index" WI, return its word, according to the global word-int mapping.

 -- Function: int bow_word2int (const char *WORD);
     Given a WORD, return its ``word index,'' according to the global word-int mapping; if it's not yet in the mapping, add it.

 -- Function: int bow_word2int_add_occurrence (const char *WORD)
     Like bow_word2int(), except it also increments the occurrence count associated with WORD.

 -- Variable: int bow_word2int_do_not_add
     If this is non-zero, then bow_word2int() will return -1 when asked for the index of a word that is not already in the mapping. Essentially, setting this to non-zero makes bow_word2int() and bow_word2int_add_occurrence() behave like bow_str2int_no_add().

 -- Function: int bow_words_add_occurrences_from_text_dir (const char *DIRNAME, const char *EXCEPTION_NAME)
     Add to the word occurrence counts by recursively decending directory DIRNAME and lexing all the text files; skip any files matching EXCEPTION_NAME.

 -- Function: int bow_words_occurrences_for_wi (int WI);
     Return the number of times bow_word2int_add_occurrence() was called with the word whose index is WI.

 -- Function: void bow_words_set_map (bow_int4str *MAP, int FREE_OLD_MAP)
     Replace the current word/int mapping with MAP.

 -- Function: void bow_words_remove_occurrences_less_than (int OCCUR);
     Modify the int/word mapping by removing all words that occurred less than OCCUR number of times.  WARNING: This totally changes the word/int mapping; any wv's, wi2dvf's or barrel's you build with the old mapping will have bogus word indices afterward.

 -- Function: int bow_num_words ()
     Return the total number of unique words in the int/word map.

 -- Function: void bow_words_write (FILE *FP)
     Save the int/word map to file-pointer FP.

 -- Function: void bow_words_write_to_file (const char *FILENAME)
     Same as above, but with a filename instead of a FILE*.

 -- Function: void bow_words_read_from_fp (FILE *FP)
     Read the int/word map from file-pointer FP.

 -- Function: void bow_words_read_from_file (const char *FILENAME)
     Same as above, but with a filename instead of a FILE*.

 -- Function: void bow_words_reread_from_file (const char *FILENAME, int FORCE_UPDATE)
     Same as above, but don't bother rereading unless FILENAME is different from the last one, or FORCE_UPDATE is non-zero.

Word Vectors
************

Creating a Word Vector from a Text File
=======================================

Writing and Reading Word Vectors as Data Files
==============================================

Vectors of Documents
********************

 -- Type: bow_dv

A Matrix of Document/Word Statistics
************************************

 -- Type: bow_dvf

 -- Type: bow_wi2dvf

Document/Word Models
********************

 -- Type: bow_barrel

Vector-per-Class Models
***********************

Arrays of Structures
********************

Arrays indexed by integers
==========================

Arrays indexed by strings
=========================

Command-line argument processing with Argp
******************************************