Borrowed from Cantonese 西梅 (sai1 mui4-2, “prune”), via Hawaii.
This etymology is incomplete. You can help Wiktionary by elaborating on the origins of this term. Particularly: “Is this Hawaiian Creole or Pidgin?”
see mui