Line5x does a couple tricks internally to generate active width of 1920/1600. For 320×240 mode, horizontal parameters are multiplied by 5, so with default settings H.samplerate=2130, H.active=1600, H.backporch=245, H.synclen=155. However, these would not be valid settings for 1920-column mode (1920+245+155>2130) so H.synclen is reduced by 120, H.backporch by 160, and H.active increased by 2*160. The final parameters thus are H.samplerate=2130, H.active=1920, H.backporch=85, H.synclen=35, H.frontporch=90.
If you set H.samplerate=386 for N64, then notice that 5*386=1930, leaving practically no time for sync & porch signals. Setting output format to 1600×1200 will solve this, but you also need to keep H.synclen>24 (same applies for all L5 optim. modes) to avoid underflowing it.