The influence of different particle size of recycled fine aggregate from clay brick (RFCB) on the surface activity was evaluated. Furthermore, the effect of particle size distribution, replacement ratio and initial water saturation of RFCB on the chloride permeability and pore structure of concrete has been carried out. The results showed that the surface activity of RFCB increased with the decrease of particle size in the alkaline environment. Compared with the reference concrete, the incorporation of RFCB increased the total pore volume and porosity of concrete, but refined the pore size distribution, thus reduced the average pore size and the most available pore size. The chloride diffusion coefficient and electric flux was reduced by 19%~59% and 24%~54%, respectively. Based on the grey correlation theory, the highest correlation, the average pore diameter and critical pore diameter were more sensitive to the chloride permeability. On this basis, the mathematical models of chloride ion diffusion coefficient, electric flux and pore structure parameters were established using the multivariate nonlinear regression analysis.