USPS - My go to place to send shipments. The best place to ship out things under 13oz. They also offer online postage (so easy drop-off or for people who hate interactions with other people *cough* me.)
UPS - Used about once. Same as Fedex IMO, but also offer online postage. Nothing wrong with them, just not my preferred place.
FedEx - I think is the most reliable, but I on'y use if sending out rather large objects for a reasonable price. No online postage.
If looking for supplies, USPS carries free envelopes/boxes that are flat rate. It's usually a better deal though if you use your own boxes for packing and would save you sometimes over $10 instead of using the USPS box depending on the size/weight. If you plan on shipping out multiple goods as a seller, ebay is a good place to buy shipping materials. $6 for about 20 envelopes is a killer compared to retail. Shipping Self-Adhesive sheets are also helpful since you don't need to tape them onto the envelope/box and has a less chance of falling off during transition. Buying a scaler would also help out with determining weight and not having to overpay during online checkouts.
I make my rounds at the post office about every other day. At USPS, if you already have a paid label, you can just drop it off into one of the specified collecting boxes if it's under 13oz.