Abstract Background Modern nursing requires a broad set of academic and practical skills, and an effective nurse must integrate these skills in a wide range of healthcare contexts. Cultivation of core competencies has recently become a key issue globally in the development of nursing education. To assess the performance of new nurses, this study developed a nursing-specific Mini-Clinical Evaluation Exercise (Mini-CEX) to evaluate the effect of postgraduate year (PGY) nurse training programs in Taiwan. Methods A nursing-specific Mini-CEX was developed based on the required core competencies of nurses. Reliability and validity were confirmed in evaluator workshops carried out prior to the administration of the pilot test and final test. Thirty-two PYG trainees were recruited with a supervisor-to-trainee ratio of 1:1.94. Data were collected from February to June 2012 and analyzed using the Kruskal-Wallis test. Results The 32 PGY trainees scored highest in the “nursing professionalism” dimension and the lowest in the “physical examination” dimension. The overall competency score was satisfactory. The trainee nurses with 19–24 months of experience scored higher than the other two groups in overall performance. Conclusion The results of this research indicate the feasibility of using our Mini-CEX tool to evaluate the competencies of PGY trainees.