In photodynamic therapy (PDT), it is desirable to determine the light fluence distribution accurately for treatment planning. Earlier studies have shown heterogeneous distribution of optical properties in patients' prostates. Finite-element method (FEM) is suitable for dealing with heterogeneous media and irregular geometries. Cylindrical diffusing fibers (CDFs) were modeled as linear sources of finite lengths, using the same parameters as those used in the treatments. Meshes were generated in the three-dimensional (3D) prostate geometry, reconstructed using transrectal ultrasound images of the prostate. Heterogeneous optical properties measured in the prostate were applied in the calculation and the refractive-index mismatch boundary condition was studied. Compared with the measurements, the FEM calculations using heterogeneous optical properties show better agreements than those using homogeneous optical properties.© (2006) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.