Hey Fish, interesting post - it set my mind to thinking about motion and language learning.
No, I haven't mastered the fine art of pen twirling, but I've definitely seen it in class; especially when teaching at middle schools in China. However, I don't necessarily see it as an indicator of boredom. Maybe you are being too hard on yourself. It could be that there is a degee of awareness on the part of students, either conscious or subconscious, that motion can assist retention. Of course, we as ESL/EFL teachers are aware of TPR as a valid method for teaching new languages - especially with young people, but I think that what we need to remember is that TPR grew from a root awareness that motion and learning are closely tied.
I used tennis balls in my middle school classes quite extensively and in many different ways. For one thing, I let my students know that I like to juggle when I get bored. So, if they started to bore me by their lack of response, I would start juggling. It worked great because it would wake me up and wake them up. I also tossed the balls around quite a bit and even encouraged them to toss them to other students during sessions where, rather than call their names when asking for a response, I would toss them the ball. In a classroom of 60+ students this would keep them on their toes due to them never knowing if the ball was coming their way or not. And interestingly enough, I found that even the less active students - those that didn't normally want to participate, actually wanted to be thrown the ball - even knowing that they would be required to use English after they caught it.
Well anyway, as I said, your post got me to thinking, so I did a quick search while drinking my morning coffee and found an interesting article online. An excerpt follows:

English through Motion and Emotion

by Jane Averill (Jane.Averill @orst.edu) & Pat Wallace

Web: http://osu.orst.edu/~averillj

English Language Institute

Oregon State University, Corvallis, Oregon 97331

Overview of Kinetic Learning Theory

In September, 1999 our English Language Institute was looking for some new ideas as we dealt with the effects of the Asian economic crisis. One of our campus consultants recommended that we listen to the ideas of Richard Warren, a retired MIT professor who had done work with Apollo guidance computer systems in the 60s and 70s. At that time, he was exposed to some accelerated learning techniques that might spark our interest. The ideas Dr. Warren put forth on that day had a great deal of intuitive appeal. He began by talking about language teaching that helped spies to perfect foreign languages so that they could pass as native speakers in 6-8 weeks. He explained that the spies learned as young children learned languages through a combination of motion and emotion. He explained some of the background for the methods and techniques that were employed by the military.

This kinetic learning has its roots in the works of Lozanov (Suggestopedia) and Rudolph Steiner (Waldorf schools). It borrows as well from the Alexander method for actors to remember lines by encoding them with movement. Certainly, Ashers TPR uses many of these ideas as well. The theory is based on the principle that using physical movement and emotional content in the classroom can accelerate learning and can help students store new concepts in long-term memory. Techniques typically employed in kinetic learning environments include music, dance, role-playing and other forms of artistic expression. John Stanford, the former superintendent of Seattle Public Schools, who became well known for turning the district around, learned of kinetic learning in officer training school while in the military. In his book Victory in our Schools (1999) Stanford refers to some of these ideas as follows:

Several decades ago, the American philosopher John Dewy pondered the best ways to combine what children learned in school with what they experienced in their lives, in other words, how to make education meaningful to them. Since Deweys time, Renate Caine, Geoffrey Caine, and other researchers in the filed of brain-based learning had found that since the brain searches for patterns and connections, and since understanding is enhanced when a lesson is presented in many different ways, an interdisciplinary curriculum helps to improve childrens academic achievement (page 32)

Teaching to appeal to multiple intelligences is something we have all heard quite a bit about. But when students sit quietly at their desks, how many parts of the brain are stimulated?

If you're interested, you can read more of the above article, and find a bibliography at: http://oregonstate.edu/~averillj/TESOL01.html