After graduating from Takigawa Daini High School, Kaji joined Cerezo Osaka in 1998. He made his first league appearance on 25 July 1998 against Avispa Fukuoka. At the club, he found it difficult to break into the first team and was loaned out to J2 League side Oita Trinita at the start of the 2000 season. After spending 2 years at Oita, Kaji came back to J1 League in 2002 to play for FC Tokyo. The manager Hiromi Hara made him a regular and assigned a more attacking role. His club won the 2004 J.League Cup by beating Urawa Reds in the final which ended 0–0 after the extra time. The penalty shoot-out was required and his successful conversion as the last penalty taker for Tokyo finally broke the deadlock. He was transferred to Gamba Osakaat the beginning of the 2006 season and scored in the beginningleague match against Urawa Reds. He was selected as a member of the 2006 J.League Best XI. The club won the champions 2007 J.League Cup, 2008 and 2009 Emperor's Cup. In Asia, the club won the champions 2008 AFC Champions League and the 3rd place 2008 Club World Cup. From 2012, his opportunity to play decreased for injury. After eight years with Gamba Osaka, Kaji signed with Major League Soccer club Chivas USA on 24 June 2014. However, the club was disbanded in 2014, he returned to Japan and joined Fagiano Okayama in 2015. He retired end of 2017 season.
National team career
Kaji was a member of the Japan U-20 national team for the 1999 World Youth Championship finals where the team finished runners-up. He gained his first cap for Japan national team when national coach Zico played him on 8 October 2003 in a friendly against Tunisia. He was rated highly by Zico who made him the first-choice right full-back. He was a member of the Japan team who won the 2004 Asian Cup in China. In a 2005 Confederations Cup against Brazil, his goal was disallowed by a controversial offside decision. His first goal for the country finally came on 17 August 2005 in a 2006 World Cup qualification against Iran. He made the final squad for the 2006 World Cup finals. In a preparation match against Germany just before the tournament, he injured his ankle when tackled by Bastian Schweinsteiger. Zico considered his replacement but decided to keep him. He missed Japan's first game in the tournament against Australia with Yuichi Komano filling in for his place, but he returned to the team and played in the rest of the competition. After the 2006 World Cup, Zico was replaced by Ivica Osim who also regularly picks him for the national team. He was a member of the Japan team for the 2007 Asian Cup and played all the Japan games in the tournament. On 20 May 2008 he announced retirement from international football. He played 64 games and scored 2 goals for Japan until 2008.