In the first sentence of the story, the narrator refers to an unreadable book. This is an allusion to a book called the Hortulus Animae, a prayer book that was popular in Germany in the 16th century:
It is well said of a certain German book that ‘er lässt sich nicht lesen’—it does not permit itself to be read. There are some secrets which do not permit themselves to be told.
The Latin title of the book, Hortulus Animae, translates to “Herb Garden of the Soul.” This, combined with the story’s theme of secret crime and the narrator’s attempts to “read” people based solely on their external appearance, suggests that the unreadable book is meant to be a metaphor for people whose “souls” cannot be read based on external clues. Although the narrator is keenly observant and attuned to clues in people’s dress and facial expressions, even he admits that not all secrets can be deduced through observation alone.
“The Man of the Crowd” is often cited as the first modern detective story, serving as an inspiration for Arthur Conan Doyle’s Sherlock Holmes and Agatha Christie’s Hercule Poirot. The narrator, like these other literary detectives, is able to deduce information about peoples’ characters and pasts by “reading” minute details in their physical appearance. The genre of detective fiction arose shortly after the creation of the first police forces in the United States and England in the 1830s. It was also written at a time when people were fascinated by criminal behavior (criminology was established as a discipline in the 19th century) and sought to predict criminal behavior based on people’s appearances. Phrenology, a pseudo-scientific theory popular in the 19th century stating that people’s personalities could be “read” in the shape of their skull, was an early example of this. Most people do not believe in these theories anymore, and phrenology in particular was often used for racist purposes in the 19th century, correlating non-white features with criminal behavior.
In short, many people in the 19th century thought that you could “read” whether or not someone was a criminal based on observation alone—and the rise of detectives in fiction, like Sherlock Holmes, exemplified this faith in the power of observation to solve crimes. The narrator’s assertion that some people cannot be read (just like the Hortulus Animae), along with the story's ambiguous ending, suggests that Poe might have been somewhat skeptical of the 19th century’s confidence in the power of observation to read all the secrets hidden in the human mind.
When the narrator first glimpses the old man, he is fascinated by his appearance, and uses an allusion—or reference to another work of art—to describe what he looks like. While the narrator sees the rest of the crowd as neatly fitting into a “type” (clerks, pickpockets, alcoholics, gamblers, etc.), the old man defies such easy categorization. Reaching for a way to make sense of his unique appearance, the narrator makes an allusion by comparing him to an image of Retzsch’s “fiend,” or devil:
[The old man’s countenance] at once arrested and absorbed my whole attention, on account of the absolute idiosyncrasy of its expression […] I well remember that my first thought, upon beholding it, was that Retzsch, had he viewed it, would have greatly preferred it to his own pictural incarnations of the fiend.
This is an allusion to Moritz Retzsch, a German artist active in the 18th and 19th centuries who was most famous for his illustrations of Johann Wolfgang Goethe’s Faust and his painting The Chess Players, which depicts a young man playing chess with the devil. While it is not clear whether Poe is here referring to Faust’s Mephistopheles, the devil whom Faust sells his soul to, or the devil pictured in The Chess Players, it is clear that the narrator is comparing the old man to the devil himself. This contributes to the narrator’s conviction that the old man has committed some unspeakable crime. As night falls, the story’s imagery becomes more hellish: gas-lamps, like hellfire, dance over the faces of the crowd, throwing them into “a fitful and garish lustre” moments before the old man appears before the narrator. As the night deepens and the narrator pursues the old man through the theater district and finally into a bar in a dangerous neighborhood, it is as if he's following the devil into the depths of hell itself.