I'm sure there are expensive consultant reports about VTA's rail line shortcomings, but Job One would be to do something about downtown, yes. Even moving the tracks to a lane on the street would help - increase the speed to at least 10 mph. Sacramento's downtown light rail actually moved noticeably better after local car traffic returned to sharing a few blocks of K Street with the trains. For both Sacramento's and San Jose's systems, the ultimate solution is some kind of subway or other grade separation, but the ridership doesn't justify the cost.

Just for fun, I looked up Sac & San Jose light rail (ranks 16 and 17 respectively) in Wikipedia by riderhip. Sacramento beats VTA quite soundly by all measures except number of stations, despite having no sales tax funding (though they're trying to get a tax started so they can afford to run more buses, and to make a better start than the present one-station-wonder - that mainly provides access to the Greyhound station that was kicked out of downtown - on a line to the airport).