Butyl stearate as phase change material was impregnated into expanded perlite, and the limestone powder was used as surface modification material to prepare perlite based phase changing aggregates(PBPCA). Then the perlite based phase changing aggregates concrete(PBPCAC) was perpared by PBPCA replacing sand. The effect of PBPCA content on fracture properties and toughness of PBPCAC was investigated through ratio of splitting tensile strength to compressive strength of concrete and fracture toughness based on three point bending beam test. Results indicate that the development of crack in PBPCAC can be defined into three stages:crack producing stage, crack stable propagation and crack unstable fracture. Compared with common concrete, PBPCAC has a longer crack producing stage and a shorter crack unstable fracture. Critical effective crack length, initial fracture toughness and unstable fracture toughness of PBPCAC are almost the same as that of common concrete. PBPCA no longer gives notable influence on concrete ductility and toughness. PBPCAC shows better fracture resistance properties.