At the northeast end of the South China Sea, tsunamis may be generated by the earthquakes along the eastward dipping subduction zone known as the Manila Trench. Tsunamis generated by major earthquakes in this zone are expected to affect Malaysian coastal areas especially in Sabah and Labuan Island. In this study, a numerical simulation of tsunami in Sabah and Labuan Island coast is conducted using a modeling code called TUNAMI-N2 (Tohoku University’s Numerical Analysis Model for Investigation of Near-field tsunami, No.2) to calculate the tsunami waveform at the output points. Two types of numerical simulations are performed in this study: (1) a numerical model based on uniform grid systems in Cartesian coordinate with 1 arc-minute and 30 arc-seconds bathymetry data and (2) a numerical model based on the nested grid system in Cartesian coordinate using non-linear theory with four different spatial grid sizes. Three scenarios of fault rupture are considered in the Manila Trench for the earthquakes with magnitudes of Mw 8.0, 8.1 and 8.5. Results of the both methods are compared and it is found that the tsunami amplitude is larger than those with use of smaller grid size of bathymetry data. However, finer bathymetry and topography data are required for better and more accurate modeling. A computational instability problem is introduced and discussed in this paper in order to obtain a better solution and stable condition in the numerical simulation. Finally, guidance for making a tsunami hazard map in Labuan Island is discussed.