I have to admit that I’m sometimes not completely sure myself. But that’s probably only the case when both is possible.
Anyways, here are some rules that might help:
1. After nouns that end in s, ß, x, z follows often es.
2. After nouns that end in an unstressed -e, -el, -er, -en, -chen, -lein, or -ling follows always s.
3. After many nouns that end in an unstressed syllable follows s.
4. After most loanwords/foreign words follows s.
The same rule applies especially to foreign words if their plural form ends in s as well.
5. After words that belong to a certain group of inflections follows s. These groups are:
- s/foreign plural, example:
6. Problem: there are many, many words that can end either in s or es. Both is possible. The list is endless, but some examples are:
As you can see, the genitive case is horribly complicated because there are almost no rules that clearly tell you what ending to use for which word. So it is actually easier to learn the genitive ending together with new nouns. It sounds like a lot of work but in the end it will save time.
[Made rebloggable on request. I deleted the old post.]