Some of the colors I use in my painting are very bad for you. Lead white, Naples yellow, lead tin yellow, and minium are all made from lead. Chromium green oxide is toxic, as is the copper carbonate in azurite, and there are many other examples. Further, all dry pigments can basically be a lung irritant if inhaled. If you are going to use dry pigments, especially the more toxic/poisonous ones, follow some basic precautions:
1) Prepare paints in a breeze-free area. The last thing you want is a gust of wind swirling up and dispersing your lead white just when you have the jar open.
2) Wash hands after painting, and change clothes or wear a smock.
3) No food, drink, or putting anything in your mouth while working.
4) Clean up work area when finished.
5) Store pigments carefully, out of reach of children, pets, and the curiously unwary (these can be adults).
There are lots of web resources for information about working with hazardous materials, as well as additional safety gear such as appropriate masks, etc.
There are also work arounds that may may not not yield as beautiful or traditionally authentic results but are safer. One example is to mix Titanium white and Zinc white in place of lead white. Titanium white by itself is a too opaque and zinc white too transparent, but mixing them is a nice compromise.