How does one go about designing phonotactics for a conlang?

Question

Most every conlanger has heard of phonotactics. However, I have yet to encounter resources about how should one go about the design of a conlang's phonotactics. How should one go about creating a language's phonotactics?

Answer 1

Though I had never heard of phonotactics for years after I started my language, I found working on phonotactics very worthwhile. I would definitely do it at the start if I began a new language.

Why did I?

In the first big iteration of my language, I had many sounds and no concepts of phonotactics. To me, it never sounded good or distinctive because of this unintentional "anything goes." I could not see or hear any a pattern or rhythm in the sounds. Learning the concept of phonotactics gave me a framework to think how I could redo my phonetics altogether and have it come out differently. I deleted and redid each lexicon entry in the new system. For me, sound is important and motivating. Once the words had some structure and cohesive identity producing aesthetic value, generating words felt less like I'm just an RNG. It's been a lot of fun. I have made up thousands more words, rather more than doubling in 2.5 years what I had done without much enthusiasm in the 8 before. In part thanks to the purposeful aesthetic and cohesiveness, I've just about always enjoyed it more than I did for any words before this sound revamp. The greater enjoyment of generating words and concepts has made world-building more engaging and more immersive throughout.

Suggestions

You could take these in steps after first reading all the bullets.

1. Read what phonotactics is in more or less detail first if you have not. 'Onset,' 'nucleus,' and 'rime' are useful at the very least.
2. Personally I am not familiar to recommend any of the large number of videos on this topic, but if you prefer that medium you might add your own useful resources answer.
3. Search videos for poetry and news broadcasts in multiple languages to find what sounds good to you. In my case, I picked one language as a phonotactical starting point, though multiple inspirations informed my language elsewhere. I would not spend too much time on it, but briefly picking a counterexample language could also be informative for the next step.
4. Study the phonotactics of your selected languages for insight and inspiration. Note how their syllables structure and don't structure in terms like CVC, CVCC, VC, etc.
5. Do you want syllable rules to change in the beginning, middle, or end of a word?
6. You're not trying to find every possible syllable in these languages. You want to recognize some patterns you like and hypothesize why. You'll test with your language.
7. Make a spreadsheet of your sounds/syllables for your reference. For instance, I noted which consonants and which vowels can appear in which position in a syllable, with the first syllable being different from mid-word syllables. In my case, I only allowed 3 consonants to "cluster" CC (e.g., my-), so I treated clusters like phonemes for spreadsheet convenience. You may benefit from a different format.

8. Start relatively restrictive to form a baseline of core syllables for your core words. As you make up more and more words and reconsider "Ok, maybe I shall allow k to end words after all" or something, you can polish in some less usual-sounding words onto your main line.
9. Brainstorm a bunch of cool-, alright-, and rule-breaking-sounding example words according to your rules. These proof of concept words don't need to be real or grammatical necessarily, just proof of concept. Shuffle and read these to test your above hypothesis. Do you like the phonotactic patterns you thought you would like? Why do you like what you like now?
10. Consider carrying on brainstorming while you're in the right gear so that when you're not you can draw on this pool of good words. I have a tinkering section at the top of my lexicon spreadsheet for words/definitions available, to-do, or up for revision, which I find helps workflow and organization.
11. Wherever you keep your lexicon (google sheets, for me), look into the tools like advanced find and replace, conditional formatting, slicers, etc. These will save you so much time if you ever decide to edit your phonotactics (or probably anything) or see how often you use certain patterns.
12. As you work on grammar or other aspects, you may now consider reserving certain phonotactical patterns for certain purposes. For examples, I make any word an adjective by repeating the first vowel and otherwise restrict duplicate VVs.
13. Make notes on how you are placing the word stress. Starting simply, develop word stress rules. You'll complicate these or realize they were more complicated already naturally as you get more used to your language.
14. Keep these phonotactical and prosodic rules both in mind and in a written reference as you devise and revise words. Sometimes you make up a nice-sounding word later to realize your pronunciation challenges your norms. When you understand what your rules are doing for you, you can understand whether to break them.

Some word-property concepts that might interest around this stage

Sonority Hierarchy

Phonostheme

Syllabic Consonant

Nonconcatenative Morphology

Vowel harmony (I don't know this site's tool at all; just the explanation was clear)

Voiceless Vowel