Yes, your cat can give you diarrhea (diarrhoea) but it is rare and the conditions are very specific. It also depends on how hygienic you are in handling cat litter and how aware you are in respect of these basic zoonotic diseases. Humans can also give you diarrhoea just for the record and the chances are much higher for a human-to-human transmission, I suspect.
Diarrhea can be caused by the protozoans (single celled organisms) cryptosporidiosis (a tiny parasite called Cryptosporidium) and giardiasis according to Cornell University College of Veterinary Medicine.
Cryptosporidiosis can also cause vomiting, fever, abdominal cramps and dehydration in cats and people. Direct contact with the faeces of domestic cats infected with ‘Cryptosporidium organisms can lead to transmission of this disease’ (i.e. transmission from cat to human).
Your cat has to have this disease first and therefore an examination of your cat’s faeces could be done by your veterinarian. The usual common sense procedures such as wearing gloves while handling substances contaminated with faeces and washing hands afterwards should prevent the transmission.
Giardia are also tiny parasites causing giardiasis. It can infect cats and other animals. It is passed in faeces. It is spread to other animals and the human-animal via surfaces, uncooked foods and water sources contaminated with giardia.
IMPORTANT: cats are rarely the source. It is normally wildlife or farm animals according to Cornell.
That pretty much answers the question succinctly. Ask your vet for more information if you are concerned. Vets should always be the central source of information of this kind.