Question: Does inline CSS and JavaScript really affect site SEO?

Answer: 
Hardly. What counts for a search engine is textual content, it will filter out everything else.

I assume CSS comes into play with search engines when they determine whether a portion of text is visible on the page, or hidden for the purposes of SEO. But even there, whether the CSS inline or in a style sheet won't make a difference.

One could argue that the more CSS and JS a page contains, the less weight do search terms have on that page. But I think that argument is invalid, because any non-content content will simply be ignored altogether.
As noted in another answer, any content created or modified by JavaScript is unlikely to be 'seen' by the search engines. Although this of course applies wherever you place the JS.

Depending on the size of your page and the capability of your hosting it could affect indexing. If the extra load is sufficient to start slowing your server's response to page requests then the search engines will slow their spidering and, for a large site, may not manage to crawl the entire site on a timely basis.