Years ago, they fled Goblin Town. WTF no fight a useless battle against their own polygon brothers. They found the secret Goblin Cave
These goblins lived in the enchanted forest for centuries. They did not fight against the Goblins from Goblintown wtf when they arrived later.
They moved to the underworld because they did not wanted to be soldiers. They preferred the powers of magical herbs.