13. Analyzing Text
There are many ways to use command-line tools to analyze text in various
ways, such as counting the number of words in a text, creating a
concordance, and comparing texts to see if (and where) they
differ. There are also other tricks you can do with text that count as
analysis, such as finding anagrams and palindromes, or cutting up text
to generate unexpected combinations of words. This chapter covers all
these topics.
13.1 Counting Text
Use the "word count" tool, wc, to count characters, words, and
lines in text.
Give the name of a file as an argument; if none is given, wc
works on standard input. By default, wc outputs three
columns, displaying the counts for lines, words, and characters in the
text.
-
To output the number of lines, words, and characters in file
`outline', type:
The following subsections describe how to specify just one kind of count
with wc, and how to count text in Emacs.
NOTE: You can get a count of how many different words
are in a text, too--see Making a Concordance of a Text. To count the average length of words, sentences, and
paragraphs, use style (see section Checking Text for Readability).
13.1.1 Counting the Characters in a Text
Use wc with the `-c' option to specify that just the number
of characters be counted and output.
-
To output the number of characters in file `classified.ad', type:
| | $ wc -c classified.ad RET
|
13.1.2 Counting the Words in a Text
Use wc with the `-w' option to specify that just the number
of words be counted and output.
-
To output the number of words in the file `story', type:
To output counts for several files, first concatenate the files with
cat, and then pipe the output to wc.
-
To output the combined number of words for all the files with a
`.txt' file name extension in the current directory, type:
NOTE: To read more about concatenation with cat, see
Concatenating Text.
13.1.3 Counting the Lines in a Text
Use wc with the `-l' option to specify that just the number
of lines be counted and output.
-
To output the number of lines in the file `outline', type:
13.1.4 Counting the Occurrences of Something
To find the number of occurrences of some text string or pattern in a
file or files, use grep to search the file(s) for the text
string, and pipe the output to wc with the `-l' option.
-
To find the number of lines in the file `outline' that contain the
string `chapter', type:
| | $ grep chapter outline | wc -l RET
|
NOTE: For more recipes for searching text, and more about
grep, see Searching Text.
13.1.5 Counting Lines per Page in Emacs
The count-lines-page function in Emacs outputs in the minibuffer
the number of lines on the current page (as delimited by pagebreak
characters, if any---see section Paginating Text),
followed by the number of lines in the buffer before the line that point
is on, and the number of lines in the buffer after point.
-
To count the number of lines per page in the current buffer in Emacs,
type:
Emacs outputs the number of lines per page of the current buffer in the
echo area.
For example, if the output in the minibuffer is
| | Page has 351 lines (69 + 283)
|
this means that the current page contains 351 lines, and point is on
line number 70--there are 69 lines before this line, and 283 lines
after this line.
13.2 Making a Concordance of a Text
A concordance is an index of all the words in a text, along with
their contexts. A concordance-like functionality--an alphabetical
listing of all words in a text and their frequency--can be made fairly
easily with some basic shell tools: tr, sort, and
uniq.
-
To output a word-frequency list of the text file `naked_lunch',
type:
| | $ tr ' ' ' RET
> ' < naked_lunch | sort | uniq -c RET
|
These commands translate all space characters to newline
characters, outputting the text with each word on its own line; this is
then sorted alphabetically, and that output is passed to uniq,
which outputs only the unique lines--that is, all non-duplicate
lines--while the `-c' option precedes each line with its
count (the number of times it occurs in the text).
To get a word frequency count--that is, the total number of
different words in a text--just pipe the output of the frequency
list to wc with the `-l' option. This counts all the lines
of its input, which in this case will be the list of unique words, one
per line.
-
To output a count of the number of unique words in the text file
`naked_lunch', type:
| | $ tr ' ' ' RET
> ' < naked_lunch | sort | uniq -c | wc -l RET
|
13.3 Text Relevance
The following recipes show how to analyze a given text for its relevancy
to other text, either to keywords or to whole files of text.
You can also use the diff family of tools to analyze differences
in text; those tools are especially good for comparing different
revisions of the same file (see section Comparing Files).
13.3.1 Sorting Text in Order of Relevance
@sf{Debian}: `rel'
@sf{WWW}: http://www.johncon.com/
Use
rel to analyze text files for relevance to a given set of
keywords. It outputs the names of those files that are relevant to the
given keywords, ranked in order of relevance; if a file does not meet
the criteria, it is not output in the relevance listing.
rel takes as an option the keyword to search for in quotes; you
can build a boolean expression by grouping multiple keywords in
parentheses and using any of the following operators between them:
| OPERATOR |
DESCRIPTION
|
| |
Logical "or."
|
& |
Logical "and."
|
! |
Logical "not."
|
Give as arguments the names of the files to rank.
-
To rank the files `report.a', `report.b', and `report.c'
in order of relevance to the keywords `saving' and `profit',
type:
| | $ rel "(saving & profit)" report.a report.b report.c RET
|
Give the name of a directory tree to analyze all files in the directory
tree.
-
To output a list of any files containing either `invitation' or
`request' in the `~/mail' directory, ranked in order of
relevancy, type:
| | $ rel "(invitation | request)" ~/mail RET
|
-
To output a list of any files containing `invitation' and not
`wedding' in the `~/mail' directory, ranked in order of
relevancy, type:
| | $ rel "(invitation ! wedding)" ~/mail RET
|
-
To output a list of any files containing `invitation' and
`party' in the `~/mail' directory, ranked in order of
relevancy, type:
| | $ rel "(invitation & party)" ~/mail RET
|
13.3.2 Listing Relevant Files in Emacs
@sf{Debian}: `remembrance-agent'
@sf{WWW}: http://www.media.mit.edu/~rhodes/RA/
The purpose of the Remembrance Agent is to analyze the text you type in
an Emacs session and, in the background, find similar or relevant
passages of text within your other files. It then outputs in a smaller
window a list of suggestions--those files that it has found--which you
can open in a new buffer.
When installing the Remembrance Agent, you create three databases of the
files to use when making relevance suggestions; when
remembrance-agent is running, it searches these three databases
in parallel, looking for relevant text. You could create, for example,
one database of saved email, one of your own writings, and one of saved
documents.
-
To toggle the Remembrance Agent in the current buffer, type:
When remembrance-agent is running, suggested buffers will be
displayed in the small `*Remembrance*' buffer at the bottom of the
screen. To open a suggestion in a new buffer, type C-c r
number, where number is the number of the suggestion.
-
To open the second suggested file in a new buffer, type:
13.4 Finding Anagrams in Text
@sf{Debian}: `an'
An
anagram is a word or phrase whose characters consist entirely
of all the characters of a given word or phrase--for example,
`stop' and
`tops' are both anagrams of
`pots'.
Use an to find and output anagrams. Give as an argument the word
or quoted phrase to use; an writes its results to the standard
output.
-
To output all anagrams of the word `lake', type:
-
To output all anagrams of the phrase `lakes and oceans', type:
| | $ an 'lakes and oceans' RET
|
To limit the anagrams output to those containing a given string,
specify that string with the `-c' option.
-
To output only anagrams of the phrase `lakes and oceans' which
contain the string `seas', type:
| | $ an -c seas 'lakes and oceans' RET
|
To print all of the words that some or all letters in a given word or
phrase can make, use the `-w' option. This outputs words that are
not anagrams, since anagrams must contain all of the letters of
the other word or phrase.
-
To output all of the words that can be made from the letters of the word
`seas', type:
This command outputs all of the words that can be formed from all or
some of the characters in `seas', including `see' and
`as'.
13.5 Finding Palindromes in Text
A palindrome is a word that reads the same both forwards and
backwards; for example, "Mom," "madam," and "nun" are all
palindromes.
To find palindromes in a file, use this simple Perl "one-liner," and
substitute file for the name of the file to check:
| | perl -lne 'print if $_ eq reverse' file
|
To check for palindromes in the standard input, specify `-' as the
file name to check. This is useful for putting at the end of a pipeline.
-
To output all of the palindromes in the system dictionary, type:
| | $ perl -lne 'print if $_ eq reverse' /usr/dict/words RET
|
13.6 Text Cut-Ups
A cut-up is a random rearrangement of a physical layout of text,
made with the intention of finding unique or interesting phrases in the
rearrangement. Software for rearranging text in random ways has existed
since the earliest text-processing tools; the popularity of these tools
will never die.
The cut-up technique in literature was discovered by painter Brion Gysin
and American writer William S. Burroughs in 1959; they believed it
brought the montage technique of painting to the written word.
"All writing is in fact cut-ups," Burroughs wrote.(21) "A
collage of words read heard overheard ... [u]se of scissors renders
the process explicit and subject to extension and variation."
These recipes describe a few of the common ways to make text cut-ups;
more free software tools for making cut-ups are listed at
http://dsl.org/comp/cutups.shtml.
13.6.1 Making Simple Text Cut-Ups
@sf{WWW}: http://dsl.org/comp/tinyutils/
To perform a simple cut-up of a text, use
cutup. It takes the
name of a file as input and cuts it both horizontally and vertically
along the middle, rearranges the four sections to their diagonally
opposite corners, and then writes that cut-up to the standard
output. The original file is not modified.
-
To make a cut-up from a file called `nova', type:
13.6.2 Making Random Word Cut-Ups
@sf{Debian}: `dadadodo'
@sf{WWW}: http://www.jwz.org/dadadodo/
No simple cut-up filter, Jamie Zawinski's
dadadodo uses the
computer to go one step beyond--it generates passages of random text
whose structure and characters are similar to the text input you give
it. The program works better on larger texts, where more subtleties can
be analyzed and hence more realistic-looking text is output.
Give as an argument the name of the text file to be used; by default,
dadadodo outputs text to standard output until you interrupt it
by typing C-c.
-
To output random text based on the text in the file `nova',
type:
This command will output passages of random text based on the text in
the file `nova' until it is interrupted by the user.
You can analyze a text and save the analysis to a file of compiled data;
this analysis can then be used to generate random text when the original
input text is not present. The following table describes this and other
dadadodo options.
| OPTION |
DESCRIPTION
|
-c integer |
Generate integer sentences (default is 0, meaning "generate
an infinite amount until interrupted").
|
-l file |
Load compiled data in file and use it to generate text.
|
-o file |
Output compiled data to file file for later use.
|
-p integer |
Pause for integer seconds between paragraphs.
|
13.6.3 Making Cut-Ups in Emacs
The dissociated-press function in Emacs makes random cut-ups of
the current buffer in a new buffer called `*Dissociation*'; the
original buffer is not modified. The text in the new buffer is generated
by combining random portions of the buffer by overlapping characters or
words, thus (usually) creating plausible-sounding sentences. It pauses
occasionally and asks whether or not you want to continue the
dissociation.
-
To generate a Dissociated Press cut-up from the current buffer, type:
| | M-x dissociated-press RET
|
Give a positive argument to the dissociated-press function to
specify the number of characters to use for overlap; give a negative
argument to specify the number of words for overlap.
-
To generate a Dissociated Press cut-up from the current buffer,
always overlapping by three characters, type:
| | C-u 3 M-x dissociated-press RET
|
-
To generate a Dissociated Press cut-up from the current buffer,
always overlapping by one word, type:
| | C-u -1 M-x dissociated-press RET
|
This document was generated
using texi2html
