Well I found a little trick with the Xbox One controller...if you have the rechargeable battery in the controller and you accidentally bump the USB cable or plug the controller for whatever reason your mouse and keyboard still work.
So I just keep that battery inside the controller while plugged into the XIM4.
A more concerning problem which I haven't seen on here yet is:
Recently I was playing Black Ops on 360 and (this has only happened twice but nonetheless annoying and it got me killed while in the middle of getting my kill streaks) the XIM4 seemed to lose connection with the 360 controller and just started blinking white and my on screen animation was even more weird as my character was slowly rotating.
The only way I could fix this is to unplug the XIM4's connection to the 360 and plugging it back in. At that point I died since I was basically standing up looking into the sky as my character slowly rotated Im pretty sure I looked like I was messing about to teammates and especially to the enemy.
Again this has only happened twice so idk if it's much of an epidemic with the software or hardware contact points but I had to bring it up since I never had any disconnection issues with my One when I owned one.
This issue has happened twice in the last week so yeah kind of annoying.
I've experienced exactly the same thing
without battery in the controller. My son thinks it's funny to walk up and unplug my controller, sometimes the xim4 still works(with the controller not plugged in), sometimes it seems to freeze the action I was last doing(if I was walking forward when unplugged, it will keep walking). However the xim4 stops responding when I plug the controller back in, requiring me to unplug the xim4. You would think with hotswapping enabled I would just be able to go on my merry way.
Bug fix? Running the newest beta/rc firmware.