The language of The Old Ones is described as "an agglutinative speech with root systems utterly unlike any found in human languages" (H.P. Lovecraft, "The Shadow Out of Time", Chapter 4). What techniques could a conlanger use to devise a unique root system for an agglutinative language?
In my opinion, it reflects, if anything, more on Lovecraft's lack of linguistic understanding than anything about the language of the Old Ones, since the roots in human languages get pretty damn ridiculous, so it's really hard to imagine something "utterly unlike any found in human languages". You would have to design a phonology produced by speech organs completely unlike those of humans to approach this.
Where I to tackle this, I would go for a system not unlike Arabic, but where the root is constituted of a set of vowels instead (with maybe "consonant harmony"?).
Alternatively, Mark Rosenfelder's Kebreni has roots where vowels are swapped around a whole lot, and I doubt that metathesis as a base derivational system exists in any human language.
Generally, I'm not sure designing something literally outside the boundaries of what the human mind can handle is by definition possible, which is what Lovecraft truly was after in his poor description attempt.
I'm not entirely sure what Lovecraft meant by "root systems", but whenever you want to do something that human languages don't or rarely do, a good first step is looking at language universals. The traditional ones, however, as set out by Joseph Greenberg, were more about syntax and only a little about morphology, which roots would probably fall under depending on what you want to do with them. Still, we can use the same techniques. We can look at what some real languages do with roots and try to figure out what they don't do, and then do that.
Roots in most languages are pretty static. They have to be, because they carry the majority of semantic content for a word, so they have to remain recognizable. Some languages, though, have morphology that alters what we would think of as the root in Western languages. In many Semitic languages, such as Arabic and Hebrew, roots are combinations of consonants, usually three but sometimes two, and inflect by changing the vowels. In Arabic the consonants k-t-b are a root that has to do with books and writing. The singular word "book" is kitāb, the plural "books" is kutub, "booklet" is kutayyib with a suffix, "writer" is kātib, and the verb "to write" is kataba.
Some languages also have infixes, which are affixes that are added inside the root instead of being added to the beginning (prefix) or end (suffix). Tagalog, for example:
For example, the verb takbo (meaning to run) + the infix um can be made into the word tumakbo, which equates to the simple past tense past tense ran.
In the Arabic example, some of the morphology was inflectional, meaning it added some extra meaning to the original root--making a plural, for instance. Some was derivational, meaning we added something to the root that created a new root word--making "books" from "book" is inflectional, but making "book", "writer", and "to write" from the root k-t-b is derivational. So one possible meaning for "root system" is a system of derivational morphology, and a "unique agglutinative root system" would be a unique agglutinative system of derivational morphology. Speakers tend to regularize morphology over time, but derivational morphology is a little more resistant to it than inflectional: in English, nobody uses "whom" anymore, but we still preserve irregularities like "in-vincible" vs. "un-beatable" vs. "a-sexual". So with derivational morphology, you have a bit more freedom to concoct a weird, wild system without having to simulate a bunch of lazy peasant speakers who can't be bothered to keep using your fancy inflections.
Anyway, hopefully this gives you some ideas for how to create your own system.