So first the short version: most TVs do some extra processing to your image in order to make it look more appealing. Though mot people don't mind, these post-processing effects are in fact adding a lot of latency to the image for mostly undesirable effects. For gaming, and especially Stadia, this is not good!!

As a result, you will want to enable your TV's "GAME MODE"!

This mode may have different names and means of being activated depending on your TV's specific model and maker. (In fact some models have even been as stupid as having "game mode" do the reverse of what it does on all other TVS)

The best solution here is to:
1- Google your TV's model number and "game mode"... you should get good results for most TVs.
2- If not, go through the settings and look for it, in many cases it has to be enabled for each output individually (in this case, your Chromecast's HDMI port).
3- Pop out your TV's manual... the info might be in there :)
4- If all else fails, you should probably read through the rant below and disable all the bad effects I mention.

When that is all done, your image may have changed quite a lot. Read the rant (and list) below if you want. But at first you might not like the result. Especially if you have been using your TVs' default settings for a while. My honest suggestion is to activate Game Mode (or at least deactivate most effects) for all your TV's content/ports. If will seem a bit dull/flat for a while, but you'll soon get used to it. And when that happens, you'll probably start seeing how much detail was being destroyed by those effects. It just takes some getting used to.

---------

OK, so now for the rant.

Modern TVs come with a huge amount of post-processing. That means it applies effects to the finished image using technology not unlike a photoshop or instagram filter (no, it's not the same, just sort of the same ball-park). And most people are fine with that. Anybody who works with digital images, makes games or does film or video mastering is likely not to be fin with these effects...

To explain this, let's go back to the reason these TVs have these effects in the first place (or at least the reason they have so many really bad effects enabled by default): when you go into a store to buy a TV, you are likely to see dozens, if not hundreds of TVs placed side-by-side. They all have the same basic features: they display images at a given size and resolution. But to differentiate their TV from the rest, brands started increasing the contrast and saturation, making the image more colorful and "vivid". This is the sort of image our brains crave, so in most cases, you'll find it more appealing and have a better chance of buying it as opposed to the other brand's model which will look dull in contrast.

The thing is, all the effect is doing is destroying a huge part of the images' details and information to "look better"... and by doing that it is no longer showing you the image that the content's creator intended for you to see. Basically, it makes the image worse but more appealing.
Now, if you really really prefer that... I guess that's fine, but don't forget, it adds a lot of latency and you will inevitably be losing some visual information.

Now... I haven't been 100% honest: some effects, though not always good for latency, are not necessarily bad. Most of those are, but some might not be. Here is a totally non-exhaustive list of effects and what they are intended to do:

GAMING-SAFE (won't affect latency):
- Brightness... seems simple.
- Contrast: I would try and leave this one at default (usually 50%), but depending on the quality of the TV, it might be worth tweaking a little bit.
- Color Balance (Red/Green/Blue and White Balance): Unless your TV really looks like the colors are off (often too red/orange or too blue/green) don't touch this.
- HDR: This adds deeper blacks and brighter whites by using various techniques and more image information, not less. It's cool!
- Sharpness (SORT OF!): The TV looks for edges in your image (edges between colors, light/dark areas, ...) and tries to accentuate them. This is kind of the reverse of FX-Anti-Aliasing (FXAA). It will make your image sharper... but can also be really destructive, making an intentionally blurry or "flat" image look like it has defined edges. I would not personally use it, but I guess it's up to you...

MEDIA-SAFE (will add latency but are not destructive for the image)
- High Refresh (has many names like Fluid Motion, High-Speed, etc.): This one is down to preference, but basically the TV will take 2 frames and try to add more frames in between based on "what the image should look like". It's a lot of trickery, and only really looks convincing when really well done, but if you're not gaming, it's fine if you like it.

DON'T USE (these add latency and will reduce visual fidelity)
- Dynamic Lighting (Sometimes Dynamic Contrast or Color): This is basically the TV trying to figure out if the image should be darker/lighter/cooler/warmer/... based on the content. What does it know? The only situation something called this is acceptable is if it is only changing the screen's luminosity using a camera (like phones do)... Not sure if that even exists.
- Color Modes (Movie Mode/Cinema Mode/Series Mode/...): These are just terrible. They make the image darker/lighter more or less contrasted/saturated to try and re-create the "style" of a given medium... But your movie is already a movie... why make it darker or more colorful? Just stick to default, you'll get the experience the content creator actually made.

For any effect you are not sure about, google it. Try and see what it actually does. If it is modifying the base image in any way, ditch it. Trust me.

---------

Now for a short story:
I once tried watching the first episode of Game of Thrones on a pretty new Samsung TV (4-5 years ago I'd say). The TV had all the default settings activated.
The first few minutes of the series are very dark, filmed at night with a pretty light blue tint to the snowy forests. On that TV, it looked like a black screen with some pretty saturated dark blue shadows moving around. There was no way of understanding what was going on at all.
The reason? High contrast, darker blacks and a whole lot of saturation. The "Cinema Mode" was active and a whole sheugh of other effects alongside it.
It may look good for the really nice mountain or forest landscape they use in the store as a demo, but when the content isn't designed for a TV (and games aren't!) it won't!