Who from the staff is in charge of selecting the music for each scene? I've been watching a lot of anime recently and realized whether the music fits a scene or not affects a lot in how much I like it, so... who's in charge of doing so?

Is it the Episode Director, the Series Director, the Sound Director or someone else? Or maybe it depends on the series?
You might want to read this article by Justin Sevakis because it really varies and depends on the type of production committee. 

TL;DR


Director or
Production Committee or
A music publisher (which is part of the production Committee) or 
An entirely separate business unit within the production Committee.



  Anime opening and ending themes are a big deal not only for fans, but
  for the creators, and the companies producing anime. Opening themes
  are a show's "best foot forward," and the staff will usually put quite
  a bit of effort into coming up with an impressive sequence for
  whatever song is chosen. Anime production committees, the group of
  companies putting up money to produce the show, often include a record
  label or music publisher, and so those theme songs are valuable places
  to promote their artists. Ads by Kiosked
  
  In some cases, the director will go into the project already wanting a
  certain existing song, or wanting to work with a particular artist to
  compose something specifically for their new project -- regardless of
  whether that artist has anything to do with the companies on the
  production committee. That's how we very occasionally get English
  artists doing opening or closing themes. Other times, the production
  committee will insist on using the songs to push a specific artist,
  and the director just has to deal.
  
  The business of anime vocal tracks is nearly always facilitated by the
  main production company or another business on the production
  committee. Many major anime producers also have affiliated record
  labels -- for example, Bandai Visual owns the record label Lantis,
  while Aniplex itself is a division of Sony Music Japan. At an early
  stage, the production committee has a meeting with the director of the
  anime, and presents him with a pile of pop songs that are in
  production. These songs are at "demo" stage -- meaning, they're not
  done, and usually have a temporary vocal track. The director usually
  just picks a song from that pile and uses it. They might tweak the
  lyrics a little bit to fit the show better. The record label will then
  go back and finish the song with the artist of their choosing, who may
  or may not be a voice actor featured in the show. If a famous
  (expensive) composer is doing the music for the show, they will often
  be asked to contribute the theme songs, as they're likely to be hits
  no matter who sings them.
  
  The business of anime songs is a huge one, and operates as a market
  separate from the rest of the Japanese music scene. The selection of
  song and artist is therefore usually a business decision.