Yes, you want to check your battery. A battery may seem OK in other regards, but still be bad/going bad… If you do not have a load tester, the cheapest way is to take it to your local big name parts store and they will test it for free.
Also, make sure your connections are CLEAN and TIGHT! I capitalize this and say will say it over and over again, because sometimes this causes the exact issue. Get a wire brush or terminal cleaner and make sure they look shiny and also ensure the terminals are connected tightly (cannot move easily by hand, preferably not at all).
Your starter could be going as well. If your battery checks out OK then check connections going to the Ground and starter. If they are good, then you may have a bad (or going bad) starter. Luckily, the starter for the ES300 is on top and easy to get to. Next time it fails to start, try hitting the starter on the metal portion with a hammer (dead blow preferred), not enough to hurt the metal, just enough to jar the starter, then try starting it. If it starts, chances are you have a worn starter. Also, you can remove it and get it tested.
Karl!