The company faces challenges that are a reminder of the early days of now-mainstream online brokerages, which suffered through untimely outages and harsh criticism from traditional finance companies and government regulators.