Kasher was born in 1895 in Warsaw, Poland (then part of the Russian Empire). His father was Rabbi Yitzhak Peretz. At the age of 19, he edited the periodical Degel Ha'Torah, the mouthpiece of the Polish branch of Agudath Israel.