If the loss function decreased, I moved the slider in the same direction. If the loss function increased, I moved the slider in the opposite direction. Using this, I was able to minimize the loss function.
I used the left and right arrow keys to accurately move the slider instead of dragging. If the slider seems stuck, I just go around the colors and repeatedly optimize the slider position until I am correct. This could take around 15 tries.
The loss function is like the altitude you are at when descending from a mountain. The lower the loss function, the closer you are to the bottom of the mountain. So as the loss function decreases and the color gets closer and closer to the target color, there is a feeling of “descending” a mountain.