During a boil water advisory, it is unsafe to consume untreated water. Taking a bath or shower is safe as long as you're careful not to swallow the water.
It's also important to be careful when bathing young children to reduce the chances that they'll swallow water.
Here is what you should do during a boil water advisory:
(Information provided by the CDC)
Boiling water
To boil water
• Fill a pot with water.
• Heat the water until bubbles come from the bottom of the pot to the top.
• Once the water reaches a rolling boil, let it boil for 1 minute.
• Turn off the heat source and let the water cool.
• Pour the water into a clean container with a cover for storage.
Disinfecting water
If you are unable to boil your water, disinfect it instead.
If tap water is clear:
• Use unscented bleach (bleach that does not have an added scent).
• Add 1/8 teaspoon (8 drops or about 0.75 milliliters) of unscented household liquid bleach to 1 gallon (16 cups) of water.
• Mix well and wait 30 minutes or more before drinking.
• Store disinfected water in clean container with a cover.
If tap water is cloudy:
• Filter water using clean cloth.
• Use unscented bleach (bleach that does not have an added scent).
• Add 1/4 teaspoon (16 drops or 1.5 milliliters) of unscented household liquid bleach to 1 gallon (16 cups) of water.
• Mix well and wait 30 minutes or more before drinking.
• Store disinfected water in clean container with a cover.
NOTE: Remember that containers may need to be sanitized before using them to store safe water.
To sanitize containers:
• Use unscented bleach (bleach that does not have an added scent).
• Make a sanitizing solution by mixing 1 teaspoon (5 milliliters) of unscented household liquid bleach in 1 quart (32 ounces, 4 cups, or about 1 liter) of water.
More information from the CDC, here.