Powerful optimization algorithms are key ingredients in science and engineering applications. In this thesis, we develop a novel class of discrete-time, derivative-free optimization algorithms relying on gradient approximations based on non-commutative maps - inspired by Lie bracket approximation ideas in control systems. Those maps are defined by function evaluations and applied in such a way that gradient descent steps are approximated, and semi-global convergence guarantees can be given. We supplement our theoretical findings with numerical results. Therein, we provide several algorithm parameter studies and tuning rules, as well as the results of applying our algorithm to challenging benchmarking problems.