In creating a language family with multiple generations, I'm noticing a huge buildup of irregularity in my inflectional paradigms, mostly caused by things like vowel loss etc. How long would it take for these strange rules of the inflection to be ironed out? Would they be at all? Is it a safe assumption to make that people will, over time, generalize the most common rule?
The way I put it in historical linguistics classes is:
- Sound laws are entirely regular, and create irregularity
- Analogy is entirely irregular, and creates regularity
In other words, neogrammarian-style sound laws apply perfectly, uniformly, and without exception across the language, and often create irregularities when they interact with each other (as you've seen). Analogy happens sporadically and unpredictably, and serves to smooth out those irregularities and make them regular again.
So the answer is yes, people will regularize things via analogy, but how quickly, how frequently, and just generally how they do that is unpredictable. Proto-Indo-European verb conjugation was a mess of irregularity; Ancient Greek preserved most of that irregularity, while Latin regularized it down to just a handful of paradigms (the five "conjugations"), plus a handful of exceptions (common words which didn't get regularized). Conversely Latin had many different ways to mark case and number on nouns, while Ancient Greek simplified these down to three (the three "declensions").
For conlanging, that means it's entirely up to you how much you want to regularize! As a general rule, the most common words are the least likely to get regularized—that's why we have odd forms like "am" and "is" and "be" in English, when practically every other verb follows a regular pattern in the present tense. But beyond this, it happens exactly as much as you the creator want it to happen, whether that's "the entire language" or "not at all".
Define "huge amount". Let's say this is for verb conjugation (maybe it's avtually for nouns; you didn't specify). If there's some commonality - e.g. vowel syncope as you mention - among enough the irregulars to number, say, half of the average size of your regular verb classes, at that point I would just call it a new verb class.
I just made up that "half of the average" metric, you could choose some other threshold, but my point is what it sounds like you're describing is a pretty common mechanism to create multiple conjugation paradigms in the first place. Then you don't have to level the irregularity if you just redefine it as regular - the regular conjugation of the "loses-its-vowel" class.
If a sound change is frequent enough, you don't even have to think of it in terms of irregular vs. regular. In Georgian, verb stems very often get a vowel or two elided. e.g. look up the word for "to kill" and you'll be told it's მოკლავს mo-k'l-av-s. The stem, which looks like -k'l-, is really an underlying -k'al-, but good luck guessing that when the /a/ gets elided in almost all conjugations... until it suddenly reappears in მოვკალი mo-v-k'al-i "I killed him". Georgian is so replete with this vowel tomfoolery that its dictionaries just don't even bother trying to keep track of which elisions are "regular" or not. There's not really a rule to determine when you do or don't elide, but calling it "irregular" belies how it's so pervasive that it resists analogical levelling.