Question: Does inline CSS and JavaScript really affect site SEO?

Answer: 
Hardly. What counts for a search engine is textual content, it will filter out everything else.

I assume CSS comes into play with search engines when they determine whether a portion of text is visible on the page, or hidden for the purposes of SEO. But even there, whether the CSS inline or in a style sheet won't make a difference.

One could argue that the more CSS and JS a page contains, the less weight do search terms have on that page. But I think that argument is invalid, because any non-content content will simply be ignored altogether.
javascript increases the site's load time. I believe that is how it may affect your site's rankings. Head over to <https://developers.google.com/speed/pagespeed/insights/> and test your URL. Sometimes using javascript on the first view of your website delays the page load as content is displayed only after the script is rendered. That may hurt user experience and SEO.