Tamoxifen treatment failure in cancer and the nonlinear dynamics of TGFβ

The process of cancer invasion involves a complex interplay between cell–cell and cell–medium adhesion, proteolytic enzyme secretion, cell birth and death processes, random and directed motility, and immune response, as well as many other factors. The growth factor TGFβ is known to have a complex effect on this process. It inhibits mitosis and promotes apoptosis in a concentration-dependent manner in vitro, and it is for this reason that its secretion is thought to be helpful in inhibiting tumour growth. However, recent in vitro and in vivo results have shown a significant effect of this growth factor in promoting the sensitivity of malignantly transformed cells to gradients of extracellular matrix proteins—an effect which tends to increase invasiveness. The drug tamoxifen has been demonstrated to be therapeutically effective in the treatment of patients with breast cancer; however, it is known also that many patients become resistant to the effect of this drug after a few years, and the reasons for this remain controversial. In this work we take our established model of cancer invasion (J. Theor. Biol. 216(1) (2002) 85), and extend it to include the effect of TGFβ. In so doing we demonstrate that a tamoxifen-stimulated upregulation of the secretion of TGFβ may give rise to a tumour which has a smaller number of cells but which has a greater invasiveness, greater metastatic potential, and a tumour histology which is known to correlate with a poorer prognosis. These data suggest that tamoxifen-stimulated secretion of TGFβ might explain treatment failure in some patients.