Based on the shear compression test results, the failure criteria of brick masonry used in representative volume element(RVE) models were proposed. Considering 3 types of brick laying pattern, 2 types of mortar bed thickness and 10 types of compressive stress level, shear compression tests for 144 brick masonry specimens were conducted using a special clamp. Considering the test results and the smoothness requirement on the failure surfaces, it showed that the Drucker Prager criteria can be used to describe the shear compression failure of brick masonry. The parameters of the criteria can be calibrated by the test results. The calibrated Drucker Prager criteria were used in RVE models, by which the compressive test and the pseudo static test of brick masonry specimens were simulated. The simulated results were in favorable agreement with the test results. The methodology and findings can be referred in numerical analysis for brick masonry materials or structures.