Why does Mika go insane? What/who causes Lain's "older sister" Mika to go insane? What happened to her (physically and mentally)?
Layer 05 "Distortion" shows how Mika goes mad (spoiler from here).

At the beginning of the episode, she is apathetic but doesn't show any symptom of mental frailty. However, she is exposed to various traumatic events:


She witnessed a car accident on Shibuya caused by the traffic lights cracking, without any reaction while all around people are scared.
She has seen her sister both in person and on a big screen on Shibuya, but Lain doesn't confirm the fact.
She got a handkerchief containing a message:





  Hell is full of the dead. and the dead shall wander.


She use the handkerchief to clean a stain caused by a boy. Later, she reads this message on a coffee stain:




  Fulfill the prophecy.


The previous traumatic events (the coffee stain, the message) are now mixed with each other while she is hallucinating. 

This is the turning point of Mika's insanity. Lain's father projection tells her how Prophecy is a way of Deus to interfere in the real world, and Mika seems to be a target of these actions, aimed to make her doubtful about her perceptions and, later, causing her dissociation. Lain can see a duplicate of her sister too in a form of a crystallized avatar at the end of the episode, and many other characters confirm the messages Mika was exposed to exist, so the first traumatic events are not hallucinations but a starter for her conditioning.

For reference, the abstract of episode 5 on Wikipedia is rough but clear:


   Lain's sister Mika is driven to the point of insanity due to the Knights repeatedly communicating the message for her to "Fulfill the Prophecy".