d3-timer keeps a reference to the callback function you pass in, so that it can call your function. Your callback might then have references to other things in-scope such as elements. If you want to dispose of a timer, you need to call timer.stop to stop it and then clear any references you have to the timer.
Thanks to @mbostock on d3 Slack.