It seems likely that the word has its origins in the French quatre, meaning four, and the Scottish wampish, meaning to wiggle, twist or flop about.
The French element, it is generally agreed, came via "quatre-corner," colloquialized as "catty-corner" and then "kitty-corner," meaning diagonally across from. Thus, the essential meaning is "to wriggle or twist in a slanted or indirect fashion."
However, in the dialect of the Deep South, a "catawampus" was also used to describe a kind of goblin or fearsome wildcat... a semi-supernatural creature of the woods. Chances are, this came from the resemblance of the word to "catamount," an archaic term for a cougar or mountain lion.
In fact, one older use of the word has it meaning "fierce" or "fearsome" -- often with the connotation of being fiercely enthusiastic. There are some lexicographers and etymologists who believe that the "wampus" element came via the Massachusett Indian muggumquomp, meaning "war chief" or "lietenant," from whence comes our more familiar "mugwump" -- a petty official, often one who cannot make up his or her mind concerning an issue.
It appears in the "uncanny creature" context in Charles Dickens' Martin Chuzzlewit (as "catawampous") in the 1840s, possibly its earliest recorded use:
"Snakes more," says he; "rattle-snakes. You're right to a certain extent, stranger. There air some catawampous chawers in the small way too, as graze upon a human pretty strong; but don't mind THEM--they're company. It's snakes," he says, "as you'll object to; and whenever you wake and see one in a upright poster on your bed," he says, "like a corkscrew with the handle off a-sittin' on its bottom ring, cut him down, for he means wenom."'
The word also, in something closer to its later meaning, served as the name (as "katawampus") of a series of Edwin Abbot novels in the 1890s.