The exponential growth of video resource necessitates the development of novel methods to facilitate video content understanding and video browsing. In this paper, we propose a novel method for browsing videos which supports user-oriented video annotation. For video annotation, we provide the semantic sketches, which include manually sketch annotation by users’ freehand drawing and sketches generated from key frames. Using semantic sketches, we not only help user better understanding the content, but also provide practical functions which facilitate the interaction with videos (1)Annotation based organization , which defines the overall structure of video clips using roughly sketch primitive symbols and hyperlinks. (2) Sketch map, which integrates semantic sketches and overall annotation in a sketch picture to visualizing video content. Finally, an informal user study shows that the presented interface enhances the accessibility and the individualization of video browsing.