Sometimes there is a fourth monkey depicted with the three others; the last one, Shizaru, symbolises the principle of "do no evil". He may be covering his abdomen or crotch, or just crossing his arms.
The main guess is that it is derived from a religious phrase, "If we do not hear, see, or speak evil, we ourselves shall be spared all evil."
It's such a good advice, especially as it is so difficult for people to hold their tongue, not backbite and gossip.
The Japanese word for monkey being "saru." However, some say that the three wise monkeys aren't Japanese in origin even though the source that popularised this pictorial maxim is a 17th century carving over a door of the famous Tōshō-gū shrine in Nikkō, Japan. The maxim probably originally came to Japan with a Tendai-Buddhist legend, from China in the 8th century.