You cannot remove certain events or clear the logs completely in any of the components. However, you can disable logging right after you enable the component. To do so, in the component settings disable the logging option. After that, the log notes that it is disabled, and the events logging stops. You can enable logging again at any time in the component settings.
Mind that you should enable and disabled the call log and SMS log separately.