(A lot of interesting work is scattered around shorter articles in scholarly journals, many of which you can find using bibliographies in the books listed here.)
Good basic introductions to principles of musical set theory
Forte, Allen. 1973. The Structure of Atonal Music. Yale. Johnson, Jeffrey. 1995. Thesaurus of Abstract Musical Properties. Westport, CT: Greenwood. — This is mostly tables of information rather than exposition on the underlying theory. Maybe less useful now that there are software tools. Compare Carter (2005) listed under “More advanced texts.” Johnson, Timothy. 2003. Foundations of Diatonic Theory: A Mathematically Based Approach to Music Fundamentals. Key College Publishing. — This book shows how basic materials of tonal music can be described using set-theoretic principles. Morris, Robert. 1991. Class Notes for Atonal Music Theory. Lebanon, NH: Frog Peak. Rahn, John. 1980. Basic Atonal Theory. New York: Longman. Straus, Joseph. 1990. Introduction to Post-Tonal Theory. Prentice-Hall (revised 3rd ed. 2004).
More advanced texts
Carter, Elliott. 2002. Harmony Book. — This wasn’t written to be a book, per se. It’s a published version, with informative commentary, of tables that this important composer compiled over a number of years for his own use during composition. Lewin, David. 1987. Generalized Music Intervals and Transformations. Yale. — A classic text. Perhaps the best balance of mathematical elegance and sophisticated musical insight. Mazzola, Guerino. 2002. The Topos of Music. Birkäuser. – Quite mathematically advanced. For most readers, this isn’t the best place to start. Morris, Robert. 1988. Composition with Pitch Classes: A Theory of Compositional Design. Yale. — This is an important text, full of suggestive ideas. More than the other books listed here, Morris’s is oriented towards composition (rather than analysis). The presentation is sometimes complex, and the printing suffers from a large number of typographical errors. If you are drawn to the sections on twelve-tone theory, then the writings of Milton Babbitt will also interest you. Morris, Robert. 2004. Advanced Class Notes for Atonal Music Theory. Lebanon, NH: Frog Peak.
Worthwhile online resource
http://www.solomonsmusic.net/setheory.htm — I’m sure there are other good resources on the web, but I haven’t made an attempt to keep current on what’s out there.
Details on Paul’s own research. (Not book-length, thankfully!)
Nauert, Paul. 2003. “Field Notes: A Study of Fixed-Pitch Formations.” Perspectives of New Music 41.1: 9–60. ——. 2003. “The Progression Vector: Modelling Aspects of Posttonal Harmony.” Journal of Music Theory 47.1: 103–124. ——. 2005. “Timespan Hierarchies and Posttonal Pitch Structure: A Composer’s Strategies.” Perspectives of New Music 43.1: 34–53. ——. 2006. “Algorithmic Strategies in A Collection of Caprices.” In Carlos Agon et al, eds. The OM Composer’s Book, vol. 1 (Editions Delatour France): 163–178. Another interesting article I mentioned in class. Hook, Julian. “Why Are There Twenty-Nine Tetrachords? A Tutorial on Combinatorics and Enumeration in Music Theory.” Music Theory Online 13.4 (2007). http://mto.societymusictheory.org/issues/mto.07.13.4/mto.07.13.4.hook.html