How fast and how casual? As you mention pizza, are you prepared to contemplate sit-down table service in an informal setting?
In any event, there is a planning law in London that stipulates that if you stand outside the front door of any branch of Pret a Manger, you must be able to see at least one other branch. The most convenient one for what you want to do is probably this one (49 Tothill St, SW1H 9LQ).
Another alternative is Farmer J St James's (8-9 Orchard Pl, SW1H 0BG), which does counter service salads (including hot salads).
A sit-down bistro-type place that I would recommend is Chez Antoinette Victoria (22 Palmer St, SW1H 0PH).
That shouldn't rule out pizza. You can get plenty of dairy-free pizza. At worst, many (probably most) pizza places will be geared up to do vegan versions, but some places might offer non-vegan but non-dairy. One possibility for a pizza would be the Pizza Express at 85 Victoria St, SW1H 0HW.