The effect of replacement of cement, sand respectively or replacement of cement and sand at the same time by fly ash on compressive strength of concrete block was studied. The results show that the compressive strength and the impermeability of concrete block are improved when suitable amount of fly ash is added. The trial production shows that, the grade of the small hollow concrete block with 16% fly ash is raised from MU 7.5, which corresponds to the concrete mix proportion used in practice, to MU 10. The permeability of block reaches the level of S grade. And saving of cement and natural sand in block are 16% and 42% respectively.