Yes, that is a realistic choice in fictive works.
Abstractly
Nobody can be 100% sure of how realistic it is, and in conlanging—or even fictive worldbuilding in general—you may always be a bit unrealistic. Think about some fictive works or conlangs you know placed in another world; they make many assumptions that extraterrestrial life has two eyes, hair on their head, ~1m–2m tall, and their mouths have lips, teeth, an alveolar ridge, a glottis and a tongue. (Your idea takes place in our (but alternative) world, but the point of the example still stands.)
Linguistically
There could be many ways of forming such a language, of which the most realistic are (chiefly from Mark Rosenfelder's Advanced Language Construction):
A creole. A creole is, by definition, a pidgin which has native speakers. A pidgin is formed when two languages try to communicate, for reasons of trade or otherwise. If you read about Tok Pisin (etymologically "talk pidgin", a creole), you'll see a lot of simplified grammar (and, as you say, English is already quite simplified).
From Swedish, but developing independently from main Swedish. As a native English speaker born in Holland, informal English I use to talk to my family is very different from main English. For example, I'd say *How late is it? (from Dutch Hoe laat is het?) by accident. Also, lots of sound changes happen. Even though I don't merge /w/ and /ʍ/, many people I know that used to be able to can't anymore, and my rhotic in both Dutch and English is [ʁ], even though neither Dutch nor the Scottish dialect of English has it.
A combination of the two. After a period of isolation (independent development), the Swedes start talking more like the Germans (like creoles, but slightly different)—a combination of the two.
So, even though it might be somewhat unrealistic, within the dome of fiction it isn't unrealistic enough to stop you.