Q:

I have question about use of "would" in this context. Is it referring to a repeated action or habit?

I took out the word would in number two. I am wondering if you could only paraphrase both number 1 and number 2, so that I may understand the use of the word would here and how using that changes the meaning of the sentence in this context.

Many years of failed and successful experimentation by many scientists followed before penicillin would be recognized as a useful medicine. It was not until after 1938 that penicillin would undergo mass production and be used worldwide as a helpful antibiotic. Penicillin was the first antibiotic discovered and is still in use today.

Many years of failed and successful experimentation by many scientists followed before penicillin recognized as a useful medicine. It was not until after 1938 that penicillin underwent mass production and be used worldwide as a helpful antibiotic. Penicillin was the first antibiotic discovered and is still in use today.

A:

Would is a 'future-in-the-past' here. The author takes up a position at the time of the discovery of penicillin and looks forward from there to years of experimentation and, eventually, mass production. Writers typically use this in order to resume the original timeframe after this "prediction".

... It would be 1938 before penicillin would undergo mass production. But in 1928 ... 

But this author is very careless about maintaining a consistent perspective, alternating simple pasts with future-in-pasts. 

Many years . . . followed before penicillin would be recognized as a useful medicine. It was not until after 1938 that penicillin would undergo mass production and be used worldwide as a helpful antibiotic. 

I think your treatment of the passage as a continuous narrative is better. However, you cannot just delete would, you must also transform the infinitive be which is its complement into the finite past form: would be →was.

Many years of failed and successful experimentation by many scientists followed before penicillin was recognized as a useful medicine. It was not until after 1938 that penicillin underwent mass production and was used worldwide as a helpful antibiotic. 

